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Executive Summary 


The Massachusetts Expanded Learning Time (ELT) initiative was established in 2005 with planning 
grants that allowed a limited number of schools to explore a redesign of their respective schedules and 
add time to their day or year. Participating schools are required to expand learning time by at least 300 
hours per academic year to improve student outcomes in core academic subjects, broaden enrichment 
opportunities, and improve instruction by adding more planning and professional development time for 
teachers. Schools draw upon state resources as well as technical assistance and support from 
Massachusetts 2020 (Mass 2020) and Focus on Results to implement expanded learning time in their 
schools. The first cohort of ten ELT schools (Cohort 1) received implementation grants to begin operating 
their expanded days in the 2006-07 school year; in 2007-08, a second cohort of nine schools (Cohort 2) 
began to implement ELT; and a third cohort of nine schools began in 2008-09, resulting in an initial 
group of 26 1 ELT schools in the Commonwealth. There has not been additional funding for new ELT 
schools since then. In the most recently completed school year, 2010-11, 19 2 schools continued to 
implement the initiative. 

Abt Associates Inc. is completing a multi-year evaluation of ELT that examines both the implementation 
of ELT in the funded schools, and the outcomes for schools, teachers, and students hypothesized to result 
from effective ELT implementation. This report describes current implementation and outcomes for an 
initiative that has been underway for five full academic years. The staggered nature of the ELT initiative 
means that as of the end of the 2010-11 school year, participating schools have completed five, four, and 
three years of implementation (Cohorts 1, 2, and 3, respectively). 


Study Design 

The overall ELT evaluation is guided by three major evaluation questions: 

1. How has ELT been implemented in schools that have received ELT grants? 

2. What are the outcomes of ELT for schools, teachers, and students? 

3. What is the relationship between ELT implementation and outcomes? 

This report addresses all three of the evaluation questions. It focuses considerable attention on how the 
ELT initiative was implemented in the ELT schools during the 2010-11 school year, and also examines 


1 In 2007-08, one Cohort 1 ELT school closed due to restructuring. Also in 2007-08, an existing Cohort 1 ELT 
school merged with a non-ELT school to become a new ELT school; in 2008-09, this new school merged again, 
this time with multiple non-ELT schools, and became a new ELT school. Given the intensity of the 
restructuring this school underwent, ESE changed this school’s cohort designation from Cohort 1 to Cohort 3. 

In 2009-10, two ELT schools, one a Cohort 2 and one a Cohort 3 school, in one district were merged; the 
combined school was designated a Cohort 3 school. During the 2009-10 and 2010-11 school years, six ELT 
schools left the initiative; in three cases the teachers’ union and in two cases the district School Committee 
voted down the school’s continued participation; in one case ESE did not renew the school’s ELT funding due 
to underperformance and because the school’s Level 4 status made it eligible for Federal SIG funding to aid 
with school turnaround. The total number of active ELT schools is as of the time of this report 19, 18 of which 
are included in this evaluation. 
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the effects of the ELT initiative on schools, teachers, and students in the three cohorts of ELT schools for 
three and four years of implementation. 2 3 Finally, the report addresses the third question through a variety 
of descriptive and exploratory analyses of variation in implementation and associated variation in 
outcomes. 

Below, the key findings from the implementation and then outcomes components are summarized. 


Key Findings 

Implementation of Core Components 

In the fifth year of the ELT initiative, all funded schools continued efforts to create a school day that 
incorporated the major elements of ELT: increased core academic time, enrichment opportunities, and 
opportunity for teachers to engage in collaborative planning and professional development. Schools 
varied considerably in their respective efforts to implement the core components. 

Core Academics and Instruction 

• The ELT school day was just under eight hours, on average, in 2010-11. 

• On average, almost five of the nearly eight hours of a typical school day were allocated to core 
academics (English Language Arts (ELA), math, science, and social studies). Twenty more 
minutes per day, on average, were allocated to core academics in 5 th than in 8 th grade. 

• Overall, the plurality of time in an ELT school day was allocated to ELA, followed by math, then 
science and social studies. Specifically, of the five hours allocated to core academics: 

- The amount of time scheduled for English Language Arts (ELA) was 1 hour and 45 minutes, 
on average. 

- Nearly 90 minutes were allocated to math instruction, on average. 

- An average of nearly 1 hour was scheduled for science and 45 minutes to social studies per 
day. 

• Time allocations for core subjects varied somewhat by grade. Specifically, 

- An average of about 45 minutes more each day was allocated to ELA in 5 th than in 8 th grade. 

- Slightly more time (12 minutes, on average) was allocated to math in 5 th than in 8 th grade. 

- About 20 fewer minutes were allocated to science and 15 minutes fewer to social studies in 
5 th than 8 th grade, on average. 

• While there are broad core principles guiding ELT implementation, ELT schools have flexibility 
in how they implement core components. As in past years, schools varied considerably in how 
they allocated time to various instructional activities. 


2 

The report presents findings based on two or three cohorts of schools in the main body; findings based solely on 
one cohort (Cohort 1) are presented, where appropriate, in appendices. 
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School-Wide Academic Focus 

• In the 2010-11 school year, most ELT schools had a school- wide academic focus, according to 
both principals and teachers. Teacher and principal reports of the focus were consistent at 1 1 of 
16 schools. 

• Both teachers and principals reported that the most common focus area was literacy, although 
writing, math, and higher order thinking skills were also common foci. 

• Elementary school teachers reported a literacy-related focus more frequently than middle schools, 
and middle school teachers reported that higher order thinking skills was the focus more 
frequently than elementary schools. 

• Students were most likely to report that the focus was math; however, student reports were rarely 
consistent with those of principals and teachers or internally consistent within a school. 

• At most schools, the focus was posted publicly, often in hallways, teachers’ classrooms, and the 
administrative offices. 

• According to principals, most ELT schools had implemented school-wide instructional practices, 
and the vast majority of teachers reported that their instructional practice was influenced by the 
focus. 

• A substantial majority of teachers reported that they used data specific to the focus area to 
monitor student progress and adjust instructional practices. Most also reported that dedicated 
academic support was influenced by the school-wide focus. 

Enrichment 

• Most ELT schools have implemented separate enrichment classes. Nearly all students participated 
in enrichment classes/instruction, though the amount of time varied. Some schools also embedded 
enrichment activities within core classes. 

• The amount of time a typical student spent in enrichment varied considerably, from daily to 
weekly. Similar to last year, middle school students appeared to spend more time in enrichment 
than elementary students. 

• Approximately half of all ELT teachers reported that they taught at least one enrichment activity. 
Middle school teachers taught enrichment more often than elementary teachers. 

• Regular teachers/staff taught some enrichment activities at most ELT schools, and over half the 
ELT schools relied on partner organizations to provide some enrichment; of those latter schools, 
regular meetings were scheduled for partner staff and teachers to collaborate, an increase over 
reported efforts to integrate partners from the previous year. 

• Most teachers reported that they and their students had some choice about selecting enrichment 
activities. The vast majority of teachers reported that all students had access to enrichment 
activities, and enrichment activities were of high quality. 

Common Planning Time and Professional Development 

• More than half of ELT teachers (65 percent) participated in collaborative planning time weekly or 
more often. 
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• Only a small proportion of teachers (16 percent) reported never having participated in 
collaborative planning. 

• Teachers reported participating in multiple activities during collaborative planning time, 
including analyzing student data, strategizing about instructional practices, and/or reviewing 
student work. The majority of teachers who reported participating in an activity also reported that 
the activity was useful. 

• Teacher perceptions of principal leadership varied across schools. Teachers were more 
consistently positive about principals’ ELT-focused leadership than they were about principals’ 
leadership in general. 

Implementation Index 

The study team developed an index keyed to core principles of effective ELT implementation, as 
articulated by ESE and Mass 2020. The purpose of the index is two-fold: one, to integrate information 
from multiple data sources into one measure that could help to describe variation in schools’ 
implementation efforts, both for individual schools and for the initiative as a whole; and two, to create a 
measure that could potentially be used to explore relationships between level of implementation and 
student achievement. 

The index is based upon interview and survey data from staff and students in both ELT and matched 
comparison schools. Its structure and thresholds reflect contributions from ESE and Mass 2020 as well as 
the study team. The index includes separate scores for eight criteria related to six dimension of 
implementation 4 , and the dimension-specific scores are also combined into an overall index score. Each 
school received a score that ranged from zero (indicating no or very little evidence of implementation on 
a given criterion) to three (indicating consistent evidence of implementation on a given criterion) for each 
of the dimensions, some of which had more than one component. 

Key findings about implementation based on application of the implementation index include: 

• The range of total scores for ELT schools was from 5 to 22, and for comparison schools, the 
range was from 2 to 12 out of a total possible score of 24. 

• The average total score for ELT schools was 11.4 and for comparison schools was 6.9. 

• ELT schools, on average, scored higher than comparison schools on six of the eight criteria. 

• Comparison schools scored notably lower on the enrichment-related criteria. 

• On average, ELT schools’ scores on individual criteria ranged from 1 to 2, and comparison 
schools’ scores ranged from 0 to 1. 

• Many comparison schools also appeared to be implementing at least some of the key components 
that are considered core expectations of the ELT initiative. 


The implementation index dimensions include: school-wide academic focus, core academics in target grades 
(two subcomponents), enrichment activities (two subcomponents), teacher leadership and collaboration (two 
subcomponents), and school leadership, along with ELT-specific stakeholder support. Chapter 3 includes 
additional details about the index. 
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• For two of eight criteria, the average comparison school score was higher than the average ELT 
score. 

Assessment of Outcomes 

Student and Teacher Surveys 

Findings from study -developed surveys are based solely on responses from the 2010-11 school year, 
regardless of individual schools’ implementation year, because the surveys were substantially revised to 
ensure more detailed information from school respondents on time use, teachers’ participation in the 
extended day teaching schedule, and overall time allocations across the schools. Consequently, survey 
responses could not be aggregated with prior survey responses to examine responses as a function of 
implementation year. 

It is also important to note that the study surveyed all teachers across all content areas, and all eligible 5 th 
and 8 th grade students in study schools, and only those schools with response rates above 70 percent for 
both student and teacher surveys are included in findings presented in the report. Teachers from 37 
schools (18 ELT and 19 matched comparison schools) participated in teacher survey administration. The 
response rates across the schools ranged from 64 to 100 percent. Eighteen ELT schools and 17 of the 
matched comparison schools achieved response rates of at least 70 percent 5 . While teacher responses can 
be assumed to be representative of teachers in study schools, student responses cannot, as they represent 
perceptions of students in only one or two grades within schools that serve between three and nine grade 
levels. 

Extant data (e.g., attendance, MCAS scores) 

Findings based on extant data sources are presented in terms of implementation year, as data were 
available for multiple academic years prior to 2010-1 1. For this latter group of outcomes, findings are 
presented in the main body for implementation years one through four (i.e., for schools with two or more 
years of ELT implementation); findings based on five years of ELT implementation (Cohort 1 only) are 
presented in the appendices. 

Analysis of extant data uses a comparative interrupted time series design that leverages pre-ELT data, 
school, and year fixed effects when estimating the effect of ELT. This design is among the strongest 
quasi-experimental designs available, although its analyses are non-experimental. Since schools and their 
students were not randomly assigned to ELT participation, results cannot be attributed solely to ELT. The 
interrupted time series design, use of matched comparison schools and statistical controls, and rigorous 
model specification, taken together, are capable of yielding credible and robust estimates of program 
impacts. This report also presents results from a number of descriptive and exploratory analyses to 
provide context for the comparative analyses; while informative, these findings do not support causal 
conclusions, as they are based upon less robust analyses. 


The two matched comparison schools that were excluded from analyses represent one Cohort 1 and one Cohort 
3 school, and both are elementary schools. 
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Non-Academic Outcomes 

Comparing Time Allocations in ELT and Matched Comparison Schools 

• The length of the ELT school day was significantly longer for 5 th and 8 th grade students than 
would be expected in the absence of ELT. 

• ELT schools allocated significantly more time for ELA, math, and science classes for 5 th and 8 th 
graders than would be expected in the absence of ELT. 

• ELT schools allocated significantly more time for non-core classes and specials for 5 th grade 
students than would be expected in the absence of ELT. 

• ELT schools allocated significantly more time for enrichment activities for 5 th and 8 th grade 
students than would be expected in the absence of ELT. 

• ELT schools allocated significantly more time for transitions, recess, snack, lunch, and 
homeroom for 5 th and 8 th grade students than would be expected in the absence of ELT. 

• A statistically smaller proportion of students in ELT schools reported that they attend an 
academic club than would be reported in the absence of ELT. 

• Significantly fewer students in ELT schools attended an after-school program than would be the 
expected in the absence of ELT. 

Teacher Outcomes 

• A significantly higher proportion of teachers in ELT schools reported that the length of the day 
allows them to accomplish their teaching goals and cover the amount of instructional material 
their students need to learn than would be expected in the absence of ELT. 

• A significantly higher proportion of teachers in ELT schools reported that they are satisfied with 
the amount of time available for instruction in ELA, math, and science than would be expected in 
the absence of ELT. 

• A significantly higher proportion of teachers in ELT schools reported that they are satisfied with 
the amount of time available for academic support, enrichment activities and for students to 
pursue topics of interest than would be expected in the absence of ELT. 

• Significantly more teachers in ELT schools reported that they are satisfied with the amount of 
time available for collaborative planning and that the length of the day allows for coordination of 
instruction than would be expected in the absence of ELT. Conversely, significantly fewer 
teachers in ELT schools than the counterfactual reported that the amount of collaborative 
planning time is a problem area. 

Student Outcomes 

• A significantly higher proportion of teachers in ELT schools reported that teachers and students 
spend sufficient instructional time together than would be expected in the absence of ELT. 

• Significantly more teachers in ELT schools reported that teacher and staff fatigue, as well as 
student fatigue, were problem areas than would be expected without ELT. Likewise, a 
significantly higher proportion of students in ELT schools reported that they were tired in school. 
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• Significantly fewer students in ELT schools reported that: they look forward to going to school; 
like being in school; that all of their classes are important to them; and that they like the length of 
their school day, than would be expected without ELT. 

• A significantly smaller proportion of teachers in ELT schools reported that student academic 
performance and homework completion rates were problem areas. 

• Students in ELT schools had statistically significantly higher suspension rates than would be 
expected in the absence of ELT ; however, while statistically significant, the differences were 
extremely small in magnitude, and therefore are unlikely to have educational or practical 
significance. 

Student Achievement Outcomes 

• In the first and second years of implementation, ELT schools served a statistically significantly 
greater proportion of minority students than estimated in the absence of ELT, although the 
estimated magnitude of the differences (3.7 and 4.0 percentage points, respectively) is unlikely to 
be practically meaningful, and there were no effects of ELT on schools’ minority student 
population in the third or fourth year. 

• In the third year of implementation, ELT schools had a statistically significantly smaller 
proportion of highly qualified core academic teachers compared to the estimated proportion in the 
absence of ELT (2.9 percentage points). 

• In the fourth year of implementation, ELT schools had a statistically significantly lower number 
of FTE teachers (4.7 fewer), and statistically significantly higher student-teacher ratio (almost 
two more students per teacher) than estimated in the absence of ELT. 

• Across all years of implementation, there were no significant differences in average student 
mobility rates between ELT and matched comparison schools. 

• Descriptive analyses restricted to ELT schools indicated variation in student performance levels 
among schools both before implementation began and in the most recent school year (2010-11), 
and indicated no consistent patterns of results. Descriptive analyses indicated that some schools 
have substantially increased the percentage of students that reached proficient or advanced 
performance levels, while others have experienced little change or decreased percentage of 
students at these same levels. 

• On average, there were no statistically significant effects of ELT after one, two, three, or four 
years of implementation on MCAS student achievement test outcomes for 3 ld , 4 th , or 7 th grade 
ELA, 4 th , 6 th , or 8 th grade math, or 8 th grade science. 

• There was a statistically significant positive effect of ELT after four years of implementation on 
the MCAS 5 th grade science test. 

• Exploratory analyses using data from non-ELT schools in ELT districts, and non-ELT schools 
statewide, rather than the study’s matched comparison schools, were generally consistent with the 
primary analysis, including the significant finding for 5 th grade science noted above. In addition, 
there was a statistically significant negative effect of ELT on 3 ld grade reading after two years of 
implementation, and there were statistically significant positive effects of ELT on 6 th grade math 
and 8 th grade science after four years of implementation in both the district-level and state-level 
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analyses. The state-level analysis also found a statistically significant positive effect of ELT on 
7 th grade ELA after one year of implementation and on 6 th grade math after three years of 
implementation. 

• Exploratory descriptive analysis linking the level of implementation in ELT schools and student 
achievement outcomes indicated no clear patterns or meaningful relationships. 

• Exploratory analysis investigating the difference of the effect of ELT in higher- versus lower- 
implementing schools indicates minimal heterogeneity in the effect by the level of ELT 
implementation. However, the effect of ELT on 8 th grade math in higher-implementing schools is 
estimated to be statistically significantly greater than the effect of ELT in low- implementing 
schools after three and four years of implementation. 


Discussion 

Across findings from interviews, surveys, and achievement data, the following themes seem clear: 

• There is strong evidence that the ELT schools have implemented many core ELT elements, both 
in terms of additional time available for instruction, academic support, and enrichment and 
supports for teachers’ use of that time. 

• There continues to be substantial variation across ELT schools’ level and approach to 
implementation (as measured by interviews, surveys, and an index). 

• Measuring different aspects of time use is challenging: collecting information on a prototypical 
student in a given grade level may or may not reveal how students are supported by the ELT 
initiative and definitions of various activities/time uses are not consistent across schools. 

• There are some, but not many, differences — even descriptively — between ELT and comparison 
schools on survey and achievement outcomes. 

• The school reform landscape is dynamic and more schools (outside of this ELT initiative) appear 
to be expanding the amount of time in their school year, as well as implementing reforms 
consistent with the core ELT components with each successive year. 

• This study was able to assess the quantity and allocation of time, but did not measure the quality 
of instruction, enrichment, and other activities made possible by the additional time, and clearly, 
the quality of such activities is also important. 


Future Steps for the ELT Initiative 

The ELT initiative has been underway for several years, and can now be considered a fairly mature 
intervention. Over that time period, as the schools’ implementation efforts have matured, the contexts 
within which the schools operate have continued to change. Some of that change reflects increasingly 
explicit guidance from the Massachusetts Department of Elementary and Secondary Education, some 
reflects the increasingly targeted nature of technical assistance and support from Mass 2020 and Focus on 
Results, and some reflects the heightened visibility of ELT more broadly, through other federal and state 
initiatives such as School Improvement Grants and Race to the Top funding. While the federal, state, and 
local contexts have changed, and the implementation of the initiative in the ELT study schools has also 
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continued to evolve, the improved academic achievement outcomes for students have not materialized as 
expected across ELT schools as a whole. 

This multi-year evaluation has described schools’ ongoing implementation efforts in four prior interim 
reports. ELT schools clearly have made progress on implementing many of the core elements of ELT and, 
as measured by the study’s implementation index, score higher than the matched comparison schools on 
average. Yet the patterns of implementation differ as much across the ELT schools as between the ELT 
and comparison schools, highlighting the variation in ELT across the initiative. The fact that such 
variation exists in the initiative’s fifth year illustrates both the complexity inherent in large-scale efforts to 
transform low-performing schools and the variation that inevitably results from flexible interventions that 
can be shared to fit individual schools’ needs. It may also reflect the different motivation of schools to 
participate in ELT from the outset, as some schools opted in voluntarily while others were strongly 
encouraged to apply. 

Variation in ongoing implementation is clearly a continuing theme for the Massachusetts ELT initiative. 
Prior years’ impact analyses have found little evidence of effects on students’ academic achievement, and 
the results from the fifth year indicate that students’ academic achievement outcomes, on average, have 
largely remained unaffected. Descriptive and exploratory analyses provide limited suggestive evidence 
that student growth in ELT schools is greater than growth in non-ELT counterparts, yet such results are 
not generally statistically significant. Despite the demonstrable progress ELT schools made to implement 
core components of ELT, those implementation efforts have not yet consistently translated the additional 
time into the content, strategies, or support that in turn yield improved overall student performance. 


Abt Associates Inc. 


Executive Summary | pg. xix 



ELT Evaluation Year 5 Final Report 


Chapter 1. Introduction and Context 


This introductory chapter first describes the existing research literature about increased learning time for 
students, and then presents a brief overview of the current federal policy context within which the 
Expanded Learning Time (ELT) initiative in Massachusetts is situated. The chapter then provides a brief 
history of the Massachusetts ELT initiative and description of the roles played by state stakeholders 
supporting ELT. Next, it provides local contextual information from the nine districts with ELT schools. 
Finally, an overview of the organization of this report is provided. 


Research Evidence about Expanded Learning Time 

Even though ELT is widely seen as a promising initiative for improving the academic achievement of 
students, there is limited rigorous research that has examined the relationship between ELT and student 
achievement. 

Variation in Expanded Learning Time (ELT) Implementation 

Though ELT refers broadly to the addition of a certain number or proportion of hours to the school year, 
schools across the country vary considerably in their implementation of ELT. Models of ELT range from 
extended academic years, to longer school days, to summer programming, to mandatory after-school 
programming staffed by adults from community-based organizations. Even within schools that are 
nominally implementing the same type of ELT, significant variation in implementation is evident. Kolbe, 
Partridge and O’Reilly (201 1) acknowledge that “more comprehensive data are needed to unpack how 
schools allocate in-school time to instruction, enrichment, and teacher professional activities.” 

Not Just Time, But Quality Time 

While ELT is believed to impact student outcomes, at least in part, by increasing the amount of 
instruction students receive, ELT and non-ELT schools alike also allocate time to lunch, attendance, 
transition periods, and various other non-instructional activities. Following the lead of a number of well- 
known education researchers (Berliner, 1990; Aronson et al., 1999), a report issued by ECONorthwest in 
2008 draws a useful distinction between types of time spent in schools. The report distinguishes between 
allocated time, the total number of hours in a school day; instructional time, the amount of time students 
are in academic classes; and academic learning time, “that precise period when an instructional activity is 
perfectly aligned with a student’s readiness and learning occurs”(Aronson, Zimmerman, & Carlos, 1999). 
Drawing on Aronson et al. (1999) and Silva’s (2007) research, the ECONorthwest authors assert that 
there is “a strong positive relationship between the amount of academic learning time and student 
achievement, a weak relationship between engaged time and achievement, but no relationship between 
allocated time and student achievement” (ECONorthwest, 2008). The important point, the paper 
continues, “is that while time in school is necessary for learning, time alone is not sufficient.” 

Indicators of and Factors Affecting ELT Program Quality 

A recent paper synthesizing current research on ELT concluded that although time itself cannot improve 
academic achievement, more time in concert with improved instructional quality was positively correlated 
with academic achievement (McREL, 2010). If using time well is integral to linking ELT with improved 
academic achievement, the question begs asking: what is quality use of time? Moreover, how can 
educators and policymakers ensure that schools maximize academic learning time? 
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As for the attributes of quality ELT programs, researchers who have studied ELT emphasize several 
characteristics. The ECONorthwest researchers note that successful ELT programs have: strong school 
leadership; involve teacher commitment and leadership; are evidence-based and data-driven; engage the 
support of parents, partners, and the community; and are focused on core academics and enrichment 
activities that are aligned with other goals and reforms (2008). Similarly, a report by Massachusetts 2020 
named “strong leaders, excellent teachers, high student expectations, careful monitoring of performance, 
and a safe, supportive, and nurturing school environment” as key to successful ELT implementation 
(Farbman & Kaplan, 2005). A RAND study of out of school time (OST) programs 6 found that quality 
programs had nine characteristics in common: a clear mission; high expectations and positive social 
norms expected of participants; a safe and healthy environment; a supportive emotional climate; a small 
total enrollment; stable, trained personnel; appropriate content and pedagogy, relative to the children’s 
needs and the program’s mission, with opportunities to engage; integrated family and community 
partners; and frequent assessments (Bodilly & Beckett, 2005). 

Several recent studies focusing on the use of expanded time in charter schools have identified both 
promising practices for ELT, as well as positive associations between ELT and improved student 
achievement in charter schools. Through examining a group of 30 high-performing expanded time 
schools, the National Center on Time and Learning identified eight practices common among high- 
performing schools: maximizing instructional time, prioritizing instruction according to an instructional 
focus, providing individualized academic support, establishing a culture of high expectations, providing a 
well-rounded education including enrichment activities, readying students for college and careers, 
working to strengthen instruction, and using data to direct instruction (Kaplan & Chan, 2011). 

A study of New York City charter schools (Hoxby, Murarka, & Kang, 2009) found that students assigned 
by lottery to charter schools performed, on average, significantly better on a number of achievement 
outcomes than students who were not assigned to charter schools. This study found correlations between 
certain achievement gains and several time-related attributes of the charter schools, including the longer 
school day and increased time spent in English Language Arts. 

Another study that compared charter schools, pilot schools, and traditional schools in Boston (Therriault, 
Gandhi, Casasanto, & Carney, 2010) found that increased school autonomy — defined by a school’s 
control of their own governance and leadership, budget, staffing, professional development, scheduling 
and time, and curriculum and instruction — was related to increased achievement outcomes. The study 
also noted that the charter schools consistently implemented a longer school day and school year than 
traditional or pilot schools, thus enabling them to provide professional development and supplemented 
curriculum. Relatedly, a study of NYC charter schools (Fryer & Dobbie, 2011) found that added 
instructional time was among five school policies that accounted for 50 percent of the variation in school 
effectiveness. Higher achieving charter schools added 25 percent or more instructional time, and showed 
annual gains in math, compared to a broader sample of charter schools. Other policies that were positively 
associated with improved student achievement included frequent teacher feedback, the use of data to 
guide instruction, high-dosage tutoring, and a culture of high expectations. 


6 It should be noted that many OST programs have important differences from the Massachusetts ELT initiative. 
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Academic Outcomes 

A recent review of research identified 15 empirical studies that examined the academic outcomes 
associated with ELT (Patall, 2010). This review found that the research designs used to look at the effects 
of expanded time generally were not rigorous enough to permit strong causal inferences; small sample 
size and other design weaknesses undermined the reliability of results. However, the results from the 
more rigorous experiments and strong quasi-experiments (such as those comparing groups of students 
through cohort or matched comparison designs) yielded more consistent results positively linking ELT to 
academic achievement (Frazier & Morrison, 1998; Adelman, Haslam, & Pringle, 1996; and Farbman & 
Kaplan, 2005). However, the association between ELT and positive outcomes varied within study 
schools, and as a function of student demographics and the specific outcome measures studied. 

Among this set of 15 studies, it is difficult to disentangle the effects of ELT on academic achievement 
from confounding factors. For example, a study conducted by the Division of Assessment and 
Accountability of the New York Board of Education (2000) found academic improvements associated 
with ELT, but also found a concentration of certified teachers in extended time schools as compared to 
non-extended time schools (Patall, Cooper, & Allen, 2010). Marcotte and Hanson (2010) also found that 
additional instructional days have similar positive effects on student achievement, but here too, other 
factors appear to be confounded with these impacts, including increased teacher quality and reduced class 
size. Patall (2010) suggested that instructional practices be “viewed as mediators of extended school time 
effects on students. That is, the effectiveness of instruction might determine whether extended school time 
has positive, negative, or no effects on student outcomes.” However, in examining the method by which 
extra time is implemented in classrooms, Patall (2010) could not determine any consistent or statistically 
significant trends. Adelman, Haslam, and Pringle (1996), for example, reported that the percentage of 
students in a Boston school who passed a state basic reading skills test increased from 77 to 90 percent 
over a three-year period. Farbman and Kaplan (2005) found that students at profiled (ELT) schools 
outperformed students of similar SES at traditional-year public schools, but later reveal that these schools 
also promoted teacher quality, strong leadership, professional development, a positive school culture, and 
family involvement, “which likely also contributed to students’ academic performance.” A thorough 
understanding of extended school time is still limited by study designs that make it difficult to isolate the 
effects of ELT on student achievement. 

Non-Academic Outcomes 

The discussion above focuses on academic outcomes. Even fewer studies have systematically examined 
the effects of ELT on such other outcomes as student engagement, social or emotional well-being, or 
student behaviors and aspirations. 

One study by the Harvard Family Research Project found that quality “expanded learning opportunities” 
for high school students improved school attendance and lowered dropout rates; improved students’ 
attitudes towards school; helped students establish stronger connections with adults and peers; improved 
students’ health and healthy choices; and provided more opportunities for learning about college and 
career options (Harris, Deschenes, & Wallace, 201 1). Another study by the Wallace Foundation noted a 
link between unstructured time for young people and “getting into trouble,” suggesting that structured 
activities contribute to students’ safety and healthy decision-making. The same study found, though, that 
access to such opportunities varied considerably by SES (Duffett & Johnson, 2004). It is of note that these 


Abt Associates Inc. 


Chapter 1. Introduction and Context | pg. 3 


ELT Evaluation Year 5 Integrated Report 


studies focused on OST, rather than ELT, though the programs of interest shared many design elements 
and objectives. 

ELT for Whom? 

Multiple studies have shown ELT to be particularly effective for low-income students. Silva (2007) 
explains that ELT might be particularly important for poor and minority students who have less access to 
educational resources outside of school. Patall (2010) found some evidence that ELT provided a 
particular benefit to “at-risk” youth (Farbman & Kaplan, 2005; McDonald et al., 2008; Ross et al., 2007; 
Meier, 2009; Van Der Graaf, 2008). These findings are consistent with those of Aronson et al, (1998), and 
Wheeler’s (1987) meta-analyses of the relevant research. A separate study conducted by Little, Wimer 
and Weiss (2008) notes that “all children, but particularly disadvantaged children, may gain a host of 
benefits that are not reflected in standardized achievement test scores but nevertheless lead to better 
overall educational outcomes. These include higher levels of engagement, higher work quality, higher 
grades, improved attendance, higher self-esteem, improved physical and mental health, lower drop-out 
rates, and lower incidence of anti-social behaviors, sexual activity and crime.” 

Areas for Future Research 

Patall (2010) suggests a number of possible directions for future research into ELT, including 
investigating the effects of ELT at different levels of schooling; including outcomes other than 
achievement in ELT studies; and examining variations in instructional strategies in the context of ELT. 
Other researchers suggest conducting a detailed cost-effectiveness of ELT study as a policy initiative in 
the context of limited resources (ECONorthwest, 2008; Kolbe, Partridge, & O'Reilly, 201 1). The recent 
attention paid to ELT as a promising strategy for improving educational outcomes is unheralded. 7 
However, given the limited rigorous research evidence about both the effects of ELT on academic and 
other student outcomes, and on the specific mechanisms/factors that would help to understand whether, 
how, and for whom ELT is effective, the need for credible evidence is clear. 


Current Federal Policy Context 

Closing the achievement gap between advantaged and disadvantaged students, especially in the nation’s 
chronically low-performing schools, is a critical national issue and the subject of focused national 
attention from federal, state, and local policymakers. Expanded Learning Time (ELT) is seen as one 
promising approach to improve the academic achievement of students, although there is limited rigorous 
research that has examined the relationship between ELT and student achievement. The schedules and 
calendars of most American public schools, defined by relatively early afternoon dismissals and lengthy 
summer vacations, is still in place from an agricultural era when children were needed at home to work in 
the fields. President Obama recently asserted that "we can no longer afford an academic calendar 
designed when America was a nation of farmers who needed their children at home plowing the land at 
the end of each day” (Obama, 2009). As persistent achievement gaps between affluent students and their 
more disadvantaged peers come under increasing scrutiny, so has the standard academic calendar. 


7 http://www.afterschoolalliance.org/PolicyFedNewsarchive.cfm;http://blogs. edweek.org/edweek/ 

beyond_schools/20 11/1 0/nclb_wai vers_touch_out_of_school_time_realm. html ;http ://ww w.ed week, org/ew/artic 
les/201 l/10/26/09elt_ep. h31.html 
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The Elementary and Secondary Education Act (ESEA), known as the No Child Left Behind Act (NCLB), 
was signed into law by President Bush in early 2002 and aimed to improve the academic achievement 
levels of public schools nationwide. Under NCLB, states must administer standardized tests to all public 
school students in grades 3 through 8 plus one year in high school to document progress towards the 
ultimate goal of having each and every student “proficient” in Math and English Language Arts by 2014. 
Schools are expected to demonstrate Adequate Yearly Progress (AYP) of their student achievement 
scores. Schools in which student achievement scores consistently miss the AYP benchmarks face 
substantial sanctions, and persistent failure to meet AYP results in special designation status often 
requires structural reforms, including closure, re-staffing, and curricular redesigns. 

Since the 2002 reauthorization of ESEA, schools have made modest progress towards the objective of 
achieving 100 percent proficiency. Indeed, in a March 201 1 address to Congress, U.S. Department of 
Education Secretary Duncan claimed that more than 80,000 of the nation’s 100,000 public schools could 
be labeled as failing under No Child Left Behind. * * 8 Of particular concern to policy makers, education 
professionals, and communities, is the persistent gap in achievement scores between students from 
different backgrounds. A report published in June 201 1 by the National Assessment of Education 
Progress (NAEP) noted that the gap between White students’ and Hispanic students’ achievement scores 
was greater than 20 points in math and 25 points in reading (Hemphill, Vanneman, & Rahman, 2011). A 
similar report from 2009 indicated that, though achievement gaps appear to be narrowing nationally, 
White students had average scores at least 26 points higher than Black students in each subject 
(Vanneman, Hamilton, & Anderson, 2009). 9 

In the current landscape of student achievement and school accountability, ELT has gained traction 
among educators and policy makers alike. In fact, in 2010, two of the four models to improve the lowest 
performing schools in the revamped federal School Improvement Grants (SIG) program require schools 
to increase learning time. The Race to the Top Fund (RTTT), which provides competitive grants to states 
to develop systemic statewide education reform, encourages expanding learning time as a strategy that 
states should seriously consider for turnaround schools. ELT is also the focus of a handful of Investing in 
Innovation (i3) Fund grants awarded by the U.S. Department of Education. According to a 2010-11 
survey conducted by the National Center on Time & Learning (NCTL), there are currently at least 1,000 
schools offering some sort of expanded schedule. The Time for Innovation Matters in Education (TIME) 
Act currently proposed in Congress would also focus nationally on expanded time and could be part of 
the upcoming reauthorization of the Elementary and Secondary Education Act. 

The notion of expanding the school day to improve student outcomes, however, is not an entirely new 
one. Historians of the ELT movement trace its beginnings to two influential reports of the early eighties 
and nineties: A Nation at Risk (1983), and Prisoners of Time (1994), which both advocated for the 
expansion of the American school day (Kolbe, Partridge, & O’Reilly, 201 1). 

Despite these earlier calls for sweeping changes to the education system, only two states currently require 
that the school year is longer than 180 days, and those that do hardly exceed the threshold (North Dakota 


In the fall of 201 1, the federal government announced a new plan for education reform that provides states the 

opportunity to apply for a waiver from ESEA accountability requirements and the flexibility to redefine 

progress benchmarks. See http://www.ed.gov/esea/flexibility. 

9 NAEP tests are scored on a 500-point scale. 
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and Ohio require 181 days and 182 days, respectively). Additionally, no state requires that students spend 
more than 1,100 hours in school a year, which, over 180 days, amounts to a six-hour school day, on 
average (National Center on Time And Learning, 2011). According to a report from Mid-Continent 
Research for Education and Learning (McREL), a non-profit education research organization, the average 
length of the school day and year do not differ at all from those challenged by the 1983 report A Nation at 
Risk (McREL, 2010). Further, a study published by the Organization for Economic Co-Operation and 
Development (OECD) found that, on average, American students spend less time in school than do 
students from other countries including the United Kingdom, Canada, China, Thailand, Russia, Korea, 
and New Zealand (OECD, 2011). Unfortunately a number of states have also been forced to reduce the 
length of the school year as of late given the difficult economic climate. 10 


History of the Massachusetts Expanded Learning Time Initiative 

The need for more learning time and the better use of this time was first formally recommended in 
Massachusetts in 1995, when the Commission on Time and Learning (the Commission) released its final 
report, Unlocking the Power of Time. The Commission was appointed to develop a plan "to extend the 
time during which students attend school" and to review the costs and implications of extending the 
length of both the school day and school year. The Commission’s report provided seven 
recommendations, including: 1) prioritizing academic learning; 2) accommodating differences in rates of 
student learning; 3) enhancing opportunities for teachers to plan lessons, participate in professional 
development activities, and collaborate with colleagues; 4) lengthening the school year; 5) providing 
optional enrichment activities throughout the calendar year; 6) using schools as Community Learning 
Centers through the day and year; and 7) promoting strong school and community partnerships. The 
Commission concluded that “time alone guarantees nothing . . . but with it, all else is possible.” 

Since its founding in 2000, the policy and advocacy organization Massachusetts 2020 (Mass 2020) has 
worked to expand learning and economic opportunities for children and families. Initially, its efforts 
focused on expanding after-school programming and summer learning opportunities; over time, the 
organization has turned its resources and attention toward the expansion and redesign of the school day 
itself. In 2005, Mass 2020 helped to launch the Expanded Learning Time (ELT) initiative, and 16 school 
districts received planning grants from the state Department of Elementary and Secondary Education 
(ESE) * 11 to explore adding time to and redesigning their school day. 

The Massachusetts ELT initiative focuses on expanding the school day and/or school year at the school 
level, and while the operational definition has changed slightly over time, the basic time requirement has 
not. Specifically, during the initiative’s first year (2006-07), this requirement meant an increase of 30 
percent more time over the prior year’s district average. The following year (2007-08), the requirement 
was adjusted to 25 percent more time or 300 hours. The requirement for the 2008-09 and subsequent 
school years was simply to expand the school schedule by at least 300 hours over the district’s average. 
During the 2010-11 school year, as in previous years, ELT implementation grants provided schools with 
$1,300 per pupil to implement their redesigned schedules. 


10 It is worth noting, however, that many charter schools, which in most states operate as autonomous self- 
contained districts operate with a longer than average school day and/or year, and the number of charter schools 
will likely continue to grow in the near future due to new federal policy and funding. 

1 1 In 2005, ESE was known as the Massachusetts Department of Education (DOE). 
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While the specific expectations for how schools should use their additional time have evolved over the 
course of the initiative, the major objectives have remained the same: 

• Provide more instructional opportunities in literacy, math, science, and other core subjects to 
support student achievement; 

• Integrate enrichment opportunities into student learning; and 

• Provide educators with increased opportunities to plan and to participate in professional 
development. 

The first cohort of 10 ELT schools (in five districts) received implementation grants from ESE for their 
expanded programs in the fall of 2006, and a second cohort of nine schools (and three new districts) 
began implementation in the fall of 2007. In the 2008-09 school year, a third cohort of nine ELT schools 
(and four new districts) began implementation, bringing the total of ELT schools to 26 12 (in 12 districts). 
There has not been additional funding for new ELT schools since then. In the most recently completed 
school year (2010-11), 19 schools (in 9 districts) continued to implement the initiative.' 3 ESE and Mass 
2020, in collaboration with other partners, continued to work closely together in a partnership to support 
the development and implementation of ELT in the funded schools. 

ESE, with support through a research grant from the Institute of Education Sciences (IES) within the U.S. 
Department of Education, has overseen a multi-year study of the Expanded Learning Time initiative to 
learn about the implementation and impact of ELT. Abt Associates Inc. is conducting this research. The 
study addresses both the implementation of the ELT initiative and the outcomes of ELT for schools, 
teachers, and students. To assess the implementation of ELT, data were gathered from school 
administrators, teachers, students, and other stakeholders through interviews and surveys. To determine 
the impact of the ELT initiative, outcomes from ELT schools (e.g., student achievement scores and non- 
academic outcomes) are compared to those from matched comparison schools. As described further in 
Chapter 2, the study controls for external policy changes by including a set of carefully chosen schools 
that are not implementing ELT. 

ELT implementation during 2010-11 was characterized by three main themes. First, the role of variation 
in ELT schools’ implementation of the initiative was recognized, as was the need to better comprehend 
and measure such variation as a means to understand the ELT model’s effectiveness. Second, 
participating schools were perceived to develop momentum in implementation at varying rates; some 


12 In 2007-08, one Cohort 1 ELT school closed due to restructuring. Also in 2007-08, an existing Cohort 1 ELT 
school merged with a non-ELT school to become a new ELT school;, in 2008-09, this new school merged 
again, this time with multiple non-ELT schools, and became a new ELT school. Given the intensity of the 
restructuring this school underwent, ESE changed this school’s cohort designation from Cohort 1 to Cohort 3. 

13 In 2009-10, two ELT schools, one a Cohort 2 and one a Cohort 3 school, in one district were merged; the 
combined school was designated a Cohort 3 school. During the 2009-10 and 2010-11 school years, six ELT 
schools left the initiative; in three cases the teachers’ union and in two cases the district School Committee 
voted down the school’s continued participation; in one case ESE did not renew the school’s ELT funding due 
to underperformance and because the school’s Level 4 status made it eligible for Federal SIG funding to aid 
with school turnaround. 


Abt Associates Inc. 


Chapter 1. Introduction and Context | pg. 7 


ELT Evaluation Year 5 Integrated Report 


were able to implement with impressive speed, and others were slower. Third, ESE and Mass 2020 have 
been both clearer and more specific about guidance and expectations for the initiative over time, and have 
implemented more formal supports; the increased levels of guidance and support were perceived to 
improve consistency and to strengthen implementation. 


State-Level Context 

Policy Context 

During 2010-11, significant state-level education reform policy initiatives continued to play a role in the 
ELT context. For example, ESE identified 34 of the state’s lowest-performing schools as Level 4 schools; 
these schools could adopt increased time as one part of a school turnaround strategy, and many of the 
Level 4 schools decided to expand their schedules. 14 Massachusetts’ Race to the Top efforts also includes 
the use of increased learning time as a school turnaround strategy for both districts and schools to 
consider. ESE provided significant support to develop and implement redesign plans in collaboration with 
the Level 4 schools, district superintendents, school committees, teachers’ unions, and other stakeholders, 
and these plans served as their application for federal grant funding. 

Roles of Key Stakeholders 

Administration of ELT Initiative 

In 2009-10 and 2010-11, there were several notable changes related to statewide oversight of and support 
for the ELT initiative. ESE engaged in a more formal accountability process than in prior years. 

Beginning in 2009-10, ESE set up multi-year performance agreements with schools (see Appendix A for 
agreement template) to demonstrate ongoing commitment by both ESE and the ELT school/district to 
rigorously implement the proposed ELT model and give the initiative sufficient time to succeed. The 
agreements were developed in collaboration with Mass 2020. The performance agreements specified 
objectives set by each participating school in three areas: improvement of academic outcomes, 
improvement of teacher leadership and collaboration, and provision of integrated enrichment 
opportunities. Each school’s agreement formed the basis of its annual reapplication. The agreements also 
provided ESE with better information with which to assess whether to continue schools’ funding; 
successful attainment of objectives mean a school will continue to receive ELT funds (assuming adequate 
funds in the state budget). Cohort 1 schools were expected to meet their performance agreement 
objectives at the end of the 2010-201 1 school year, and Cohort 2 schools will need to attain their 
objectives by the end of the 201 1-2012 school year. The increased attention to actual performance is 
reflected in greater clarity of and accountability for expectations over time, and are intended to improve 
(at both individual school and initiative levels) the quality of implementation. However, ESE staff support 
for the ELT initiative decreased in 2010-11, due to turnover and agency reorganization transitions, 
leaving one staff person with primary responsibility for coordinating ELT-related support. 

Policy/Advocacy and Implementation Support 

Mass 2020 is a policy, advocacy, and technical assistance organization focused on developing and 
implementing both a national and state agenda for expanding learning time and redesigning schools. 
During 2010-11, Mass 2020 remained a strong advocate for the initiative and provided targeted technical 
assistance to ELT schools. 


14 None of these Level 4 schools are among this study’s sample of ELT schools. 
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Technical assistance to schools was also further formalized during 2010-11. Mass 2020 took the lead for 
organizing and facilitating regular school leadership sessions, and also provided coaching to a subset of 
schools in collaboration with Focus on Results. Focus on Results is a national consulting organization that 
works with schools and districts on plans to improve performance, leadership, and decision-making. 

ESE and Mass 2020 continued to collaborate closely as key state partners to support the development and 
implementation of the ELT initiative. Jointly developed by ESE, Mass 2020, and Focus on Results, the 
ELT Expectations for Implementation 15 continued to help define “good work” in an ELT school and 
guided technical assistance during 2010-11 (see Appendix A). The Expectations were divided into seven 
areas with corresponding indicators to detail associated practices. These seven areas included: 1) clear 
school-wide academic focus; 2) additional time for core academics; 3) additional time for enrichment; 4) 
additional time for teacher leadership and collaboration; 5) focused and collaborative leadership; 6) 
alignment and focus of resources; and 7) district leadership for ELT. 

During 2010-11, 14 funded ELT schools participated in formal technical assistance through coaching 
and/or leadership training sessions. Mass 2020 provided coaching to six schools, and Focus on Results 
provided coaching to an additional five schools. Three schools attended leadership training sessions but 
were not assigned a coach. 

Coaching played an important role in 2010-1 1 because it allowed for more differentiation in the support 
of schools. Coaches helped schools with their performance agreements, monitored progress on ELT 
implementation, and assisted with individual school priorities for improvement. The format ranged from 
one-on-one time with the principal or the principal and ELT coordinator to time with the entire 
instructional leadership team. In addition to in-person time, the coaches stayed in touch with schools via 
phone and e-mail throughout the year. 

The approach to coaching, in general, continued to involve first focusing on a school’s instructional 
approach, next coaching the school in practices for teachers aligned with this approach, and then focusing 
the school on specific professional development and planning time that are linked to these practices. Next, 
the approach involved focusing on data and information gathering linked to the practices, encouraging 
public use of the data, and focusing the school leadership team on owning and leading the improvement 
process. 

During 2010-11, all schools that participated in the technical assistance process focused multiple topics 
including strategies for the best use of the additional time, how to design academic support periods, the 
use of time analysis tools, and additional layers of support that schools might consider including summer 
and weekend instruction. Since performance agreements were finalized with ESE in 2009-10, schools 
spent extensive time working towards these goals during the 2010-11 school year and thinking about how 
best to use data to measure academic progress. Bimonthly school leadership team sessions designed and 
facilitated by Mass 2020, focused on teacher collaboration, ensuring that appropriate protocols and 
meeting structures were in place, and that student data were analyzed at these meetings. 


15 See Appendix B for ELT Expectations for Implementation. 
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Both Mass 2020 and Focus on Results provided ESE with additional capacity to both build support for the 
initiative and directly support participating schools with implementation. ESE also did not have the staff 
capacity to provide coaching and intensive implementation support to ELT schools, and having partners 
in this role added valuable expertise in this area. State stakeholders also reported that it was helpful to 
have a non-governmental entity serving in the coaching role to lend authenticity to these relationships and 
separate the government’ s accountability role from ongoing interactions with coaches. 

Funding 

In fiscal year 201 1 (July 1, 2010-June 30, 201 1), the state appropriated approximately $13.9 million for 
ELT grants. The fact that this line item remained essentially level (although slightly lower compared to 
fiscal year 2010) was a testament to the Legislature, Governor, Mass 2020, and ESE’s support of the ELT 
initiative. 

ESE provided grants to 19 schools in 2010-11 at a level of $1,300 per student; this per-student amount 
has remained the same for all four years of the initiative. Grant amounts ranged from approximately 
$235,000 to just over $1.5 million per school, reflecting variation in ELT school enrollment from 181 to 
1,167 students. Preliminary grant amounts were calculated using estimates based on the school’s prior 
year enrollment, and several adjustments to final grants were made when enrollment varied from 
estimates by more than ten percent. Funds for planning grants were not available during 2010-11 for the 
third consecutive year. 

In a report detailing the allocation of Massachusetts’ ELT grant funds (O’Reilly & Kolbe, 201 1), a fair 
amount of variation was found in the amount that each school spent, per-pupil, on the implementation of 
ELT. A number of schools reported expenditures substantially higher than the $1300 received from the 
state grant. In every ELT school, most spending went towards instructional costs (77% on average). On 
average, schools distributed the balance of their grant funds relatively evenly across administrative costs; 
transportation facilities and meals; and supportive services (between seven and ten percent each), though 
there was variation between schools within categories. Four district respondents reporting data about nine 
schools indicated that they used funds other than state grant funds to support ELT in their schools. For all 
nine schools, the ELT grant funds provided the largest share of funding. 


School District Context 

The ELT schools in 2010-11 were located within nine school districts. During the spring of 2011, Abt 
Associates study team members interviewed district-level staff at eight of the nine Massachusetts districts 
of which ELT schools are a part. All interviews covered the same topics, including the presence of ELT 
practices and indicators at non-ELT schools, district supports for ELT, and other district policy and 
program initiatives that affect, replicate, or compliment the implementation of ELT in the district. 

Overall, these interviews revealed that many of the core components of ELT, namely extra time for core 
academics, academic support, enrichment, common planning time for teachers, and a school-wide 
academic focus, are implemented at some or many schools that do not receive ELT funding. ELT schools 
were reported to benefit from additional support structures including technical assistance on 
implementation, financial support for extra staff, and the ability to implement programs under one roof 
without requiring that students travel to a separate site. The following section describes the 


pg. 10 | Chapter 1. Introduction and Context 


Abt Associates Inc. 



ELT Evaluation Year 5 Final Report 


implementation of ELT across all schools in participating districts, as well as districts’ support for ELT, 
and other policy initiatives that may have affected ELT implementation. 

Extra Time 

In five of the eight districts in which interviews were conducted, district staff noted that some of the 
district schools not formally implementing ELT had added time to the school day. It is important to note 
that there were typically only a few such schools in each district, except in one large urban district, in 
which more than a few schools expanded their days. Though district staff generally expressed enthusiasm 
for ELT, they noted certain necessary conditions needed to exist for schools to add time to the school day. 

The primary barrier to expanded time was funding. The district schools that added time without ELT 
funding did so either using money from the level 4 turnaround grants or innovation grants. Additionally, 
some of the schools that extended their days were pilot schools which had increased flexibility around 
budget allocation. Another consideration highlighted by district staff was staffing and teacher contracts. 
One of the three districts that did not add time at any additional schools cited collective bargaining as the 
barrier. One school that did add time to their schedule, despite teacher contract issues, did so by 
staggering teacher start and end times, so that no teachers worked additional hours yet the school was 
operating for additional time. 

Academic Support 

When asked about academic support practices in district schools, interviewees from the majority of 
districts noted that some or all district schools provided some level of academic support to students. 
Though asked specifically about providing dedicated blocks of time for academic support, respondents 
also described pull-out, tutoring, and paraprofessional supports. It is unclear from interviews whether 
other schools were providing academic support to all students in the same fashion that ELT schools 
tended to do so, namely, with a dedicated block of time for all students. One district noted that all 
struggling students received a dedicated block of academic support. Within districts, schools varied in 
their design and implementation of academic support programs, though across districts, respondents 
named SES and Title 1 grants as the means by which these programs were funded. 

Only one district indicated that they did not provide academic support in any of the non-ELT schools. “It 
is something that we have been talking about,” the respondent noted, but “the staffing piece of academic 
support is somewhat challenging for schools that do not have the extra time”. 

Enrichment 

Enrichment programming was less- widely implemented than academic support in districts’ non-ELT 
schools. Only one district noted that enrichment activities were provided in every school in the district. It 
is notable that this district is a particularly small one. In all other districts, interviewees noted that some 
schools had arts or athletic programs that might qualify as enrichment, but that few schools had them, or 
if so, few students participated. By and large, enrichment programming seemed to be confined to ELT 
schools and voluntary after-school initiatives at non-ELT schools in these districts. 

Interestingly, all eight district representatives indicated that their districts had at least some after school 
programs and opportunities, especially in the elementary and middle grades. Five district interviewees 
indicated that all schools in the district had after school programs, while in other districts, these programs 


Abt Associates Inc. 


Chapter 1. Introduction and Context | pg 1 1 


ELT Evaluation Year 5 Integrated Report 


were limited to Title 1, or another subset of schools. Three districts explained that 21 st Century 
Community Learning grants enabled them to provide after school opportunities. 

Across the board, after school offerings were buoyed by partnerships between schools and community 
organizations. In all but one district, where the teachers’ union prevented outside community partners 
from being paid to work in the schools, interviewees reported that there was abundant enthusiasm and 
interest among organizations for partnering with the district. Though partnerships with community 
organizations were active in all district schools, interviewees noted that ELT schools had more experience 
and stronger structures for capitalizing on these partnerships. “ELT has served as a community model in 
terms of how schools and community organizations can access each other and work together,” noted one 
respondent. 

Common Planning Time 

Common planning time is another aspect of ELT that, according to district interviewees, occurs 
ubiquitously across all schools in all districts, though interviewees noted that the frequency and duration 
of these meetings varied considerably. Several respondents explained that ELT schools had led the way 
for the development and implementation of collaborative planning time in non-ELT district schools. 
Others noted that common planning time was a district-wide focus, or a strategy that had been in place 
since before ELT. 

School-wide Academic Focus 

Similarly, the identification of a school-wide academic focus occurred in most or all schools, not just ELT 
schools. One district noted that this practice had been in place since 2003. District respondents identified 
reading comprehension and the instructional core as specific examples of academic foci. A school’ s 
academic focus often included or complemented an identified set of common instructional practices. In 
two districts, respondents did not mention specific academic foci, but identified instead, commonly used 
instructional programs like the Bay State Reading Initiative. Though school-wide academic focus and 
common instructional practice are distinct from one another, they can be related aspects of school and 
district culture. 

District Support for ELT 

District respondents revealed that the district’s roles and responsibilities in supporting ELT varied. In 
general, the amount of leadership and guidance required by schools from the district appeared to wane 
over time, requiring more support initially, and less each subsequent year of implementation. This pattern 
was untrue, however, in at least one case in which the district became increasingly involved in later years, 
as ELT grant funding was jeopardized by the schools’ poor academic outcomes. 

The most commonly mentioned district supports for ELT were administrative in nature. District 
interviewees noted that they helped develop the grant application, negotiated expanded teacher contracts, 
and navigated logistical issues related to transportation and food. One respondent noted that district staff 
attended ELT technical assistance meetings with ELT schools’ administrative teams, but this practice did 
not seem common across districts. Beyond initial grant, contract, and logistical supports, district 
interviewees explained that ELT schools received many of the same supports that non-ELT schools 
received, which varied across districts. Smaller districts generally reported more involvement with and 
knowledge of their schools than did the larger districts. 
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School Context 

Interview and survey data provided useful information about several contextual factors that influence 
ongoing ELT implementation in study schools, including principal mobility, scheduling and staffing, 
teacher leadership, key priorities, and out of school programming. 

Principal Mobility 

ELT implementation may or may not necessarily be affected by principal turnover, although one might 
well assume that changes in school leadership influence the overall school climate and staff experiences. 
Over the course of the study period, 2006-07 to 2010-11, ten ELT schools had stable principal leadership, 
six ELT schools experienced one principal turnover, one school experienced two principal turnovers, and 
another school experienced three principal turnovers. 

Staffing 

Schools varied with respect to how much of the expanded time schedule was covered by classroom 
teachers. According to principal interview data, in 2010-11, teachers from every ELT school worked at 
least some portion of the expanded hours. In the majority of ELT schools (17 of 18), more than half of 
the teachers worked the entire expanded schedule, and in two-thirds of ELT schools, 90 percent or more 
of teachers were working for the entire expanded day schedule. Teacher survey data indicate consistent 
results: 65 percent of teachers reported working the entire expanded day, 23 percent reported working 
some of the expanded hours, and eight percent of teachers reported working no expanded hours. 

Teacher Leadership 

The majority of teachers (59 percent) agreed that they (teachers) were involved in key decision-making at 
their schools; all ELT principals described the school-based Instructional Leadership Team (ILT) as a 
commonly used vehicle for such teacher participation in decisions. The typical membership included 
teachers (in all 18 ELT schools), principal (in 16 schools), and other administrators (in 12 schools), and 
schools varied in whether Title I teachers, special education teachers, and/or guidance counselors 
participated in ILTs. During the school year, all ILTs met at least monthly, and 75 percent met at least 
every other week. The meetings agendas were established primarily by the principal in slightly more than 
half of the schools (10 of 18), and by all or rotating team members in the remaining eight. The primary 
mechanisms for disseminating information from ILT meetings were the posting or emailing of meeting 
notes (nine ILTs) or teachers from the ILT would report back to their teams (six ILTs). 

Key Priorities 

Principals were asked to describe their school’s key priorities for the 2010-11 school year. Every principal 
included academic topics in their descriptions, with considerable variation in the accompanying level of 
detail provided. The majority (13 of 18 principals) reported that ELA was a key priority, and seven of 
these principals specifically emphasized a focus on improved ELA MCAS scores. Slightly fewer than half 
(seven of 18) principals reported that math was a key priority, and five referred specifically to improving 
math MCAS scores. A few principals listed instructional strategies as their priorities (4-square writing, 
QAR, Two-column notes). Other key priorities reported by a handful of principals were improving 
enrichments (3 principals), professional collaboration (2), and student discipline (2). 
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Additional Out of School Programming 

In addition to the expanded school day, most ELT schools (16 of 18) offered some sort of activities 
beyond the school day. Principals reported on their schools’ after school, Saturday school, summer 
school, and vacation week programming. Two-thirds of schools (12 of 18) reported that they offered 
after-school programming (e.g., tutoring, homework help, and sports), Saturday school, and summer 
school. Across these types of out of school time, the activities offered were primarily academic in nature, 
and included academic support for low-performing students, homework help, and language skills practice. 
Fewer schools (7 of 18) offered organized activities during vacation weeks. Field trips, accelerated 
academics, and core subject work were some examples of vacation week programming. Federal funds 
provided support for additional out of school programming for about half of the ELT schools (8 of 18). 
Other sources include private fundraising (two schools) and district funds (two schools). One school 
reported using its ELT grant to fund these extra programs. 


Organization of this Report 

Study findings on both ongoing implementation and outcomes from the 2010-11 school year, the fifth and 
final year of the evaluation, are presented in this integrated report. Chapter 2 describes the study design, 
data collection methods, and analytic approach. Chapter 3 describes an implementation index developed 
to measure aggregate ELT implementation and its application. Chapter 4 presents detailed 
implementation findings for ELT schools in 2010-11. Chapter 5 describes non-academic outcomes 
findings, and Chapter 6 describes student achievement outcomes findings. Finally, Chapter 7 includes 
conclusions and recommendations for future policy development and research. 
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Chapter 2. Study Design 


The findings in this report are based on data collected during the 2010-11 school year. While the general 
framework for the study design has not changed over the course of this five year study, selected features 
have changed to accommodate schools’ entry into the ELT initiative in multiple cohorts, changes in the 
analytic sample because some schools exited the initiative, and lessons learned from prior years that 
informed changes in data collection instruments and strategies. This chapter begins with a description of 
the conceptual model that underlies the initiative and its evaluation. It then describes the core research 
questions, quasi-experimental study design, and study sample, including how comparison schools were 
selected for the strongest possible study design. The discussion then turns to a description of the data 
collection measures, procedures, and response rates before outlining the analytic approaches used to 
describe implementation and estimate outcomes. 


Conceptual Model of the Massachusetts ELT Initiative 

This evaluation is based on an understanding of the Massachusetts ELT initiative’s components and 
intended outcomes; the conceptual model is displayed in Exhibit 2.1. The first step in successfully 
redesigning schools to provide expanded learning time requires extensive planning as well as the ability 
to overcome numerous logistical and political obstacles, such as balancing adequate time for additional 
instruction, enrichment opportunities, and teacher planning and coordination. Districts and schools rely on 
the cooperation of community stakeholders who may help facilitate (via resources or political 
connections) or who may impede the planning process. 

As shown under “Key ELT Components,” schools that successfully implement these core ELT tenets are 
expected to provide three major components: (1) more instructional opportunities in core subjects for 
students; (2) increased enrichment opportunities that engage students in learning; and (3) increased 
opportunities for educators to plan, collaborate, and participate in professional development opportunities. 
It should be noted that a number of mediating factors may affect an ELT school’s ability to successfully 
implement these core components. Schools may differ in the degree and quality to which they implement 
the three core factors. For example, there may be important school or district contextual factors, such as 
parental support for the initiative or the ability of schools to leverage the involvement of partner 
organizations, which affect how schools implement ELT’s main components. 

When implemented successfully, these “Key ELT Components” are hypothesized to affect a number of 
desired outcomes, particularly for students and teachers. For example, theory suggests that as a result of 
ELT, students may become more engaged in school because there are additional enrichment 
opportunities. Similarly, they may develop better communication and problem-solving skills as a result of 
more time with teachers and peers, and, with less idle time available, become less likely to engage in 
disruptive behavior. Teachers may find their teaching experience more rewarding and satisfying because 
they have adequate time to plan, collaborate, and instruct, as well as earn higher pay and have additional 
opportunities to develop meaningful relationships with students and peer educators. Parents too may be 
more satisfied with their children’s schooling experience. Ultimately, improvements in these outcomes 
may lead to increased student achievement. 

This evaluation examines each element presented in the conceptual models in turn. The discussion began 
in Chapter 1, which focused on the history, planning, and challenges in implementing ELT in broad 
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terms. The evaluation team next examines the implementation of the “Key ELT Components,” beginning 
with a discussion of the degree to which ELT schools have addressed key implementation components 
(Chapter 3), before moving into a more detailed exploration of each of these items (Chapter 4). Chapter 5 
examines the relationship between ELT, teacher outcomes, and student non-academic outcomes. Chapter 
6 addresses the final item in the conceptual models, assessing the impact of ELT on student achievement 
over the course of the study period. 


Exhibit 2.1: Conceptual Model of Expanded Learning Time (ELT) Initiative in Massachusetts 
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Key Research Questions and Study Design 

Key Research Questions 

The research questions of the ELT evaluation are guided by the conceptual model presented in the 
previous section as well as additional key objectives that have evolved over the course of the initiative. 
The research questions are as follows: 

Research Question 1: ELT Implementation 

1. To what degree have the “Key ELT Components” been implemented? 

2. How have the “Key ELT Components” been implemented? 

3. What are the opinions and attitudes of key stakeholders about the quality and effectiveness of 
ELT? 
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Research Question 2: ELT Outcomes 

After three or more years 16 of implementation of ELT, is there a relationship between ELT and: 


1. Non-academic outcomes 

a. Instructional and non-instructional time 

b. Teacher outcomes 

i. Satisfaction with school and district leadership 

ii. Satisfaction with time available for instructional activities, planning and professional 
development 

iii. Job satisfaction 

c. Student outcomes 

i. Student-educator relationships 

ii. Student satisfaction and engagement 

2. School characteristics and academic outcomes 

a. School characteristics and student mobility 

b. Student Massachusetts Comprehensive Assessment System (MCAS) scores 17 
Research Question 3: Implementation and Outcomes 

1. Is there an observable descriptive relationship between the percent of a school’s students reaching 
the proficient or advanced MCAS levels and ELT implementation; and 

2. Is there a difference in the effect of ELT on MCAS scores by level of ELT implementation in 
schools? 

This final report focuses on the above questions for the three cohorts of ELT schools, which have been 
implementing the initiative for up to four years. 18 

Study Design 

All of the analyses described below are non-experimental, since the random assignment of schools or 
students to ELT was not feasible. Although ideally one would like to be able to attribute any observed 
differences between the groups on study outcomes to participation in ELT, the results from non- 

16 As of 2010-11, all schools in the study have implemented ELT for at least three years. 

17 The annual MCAS assessment tests all public school students in Massachusetts in English Language Arts 
(ELA) and Mathematics in grades three through eight, and grade ten and in Science and Technology/Education 
in grades five, eight, and nine through twelve in order to measure performance based on Massachusetts 
Curriculum Framework learning standards. See http://www.doe.mass.edu/mcas/ for more information about this 
assessment. 

Is In the main body of the report, findings are presented after one, two, three, and four years of implementation 
because only the schools who began implementing ELT in 2006-07 have implemented the initiative for five 
years. Supplementary exhibits with the preliminary findings after five years of implementation are presented in 
Appendices D and E. 
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experimental studies generally do not permit reliable inferences because reasonable alternative 
hypotheses can be offered to explain observed differences, other than participation in ELT. The most 
common alternative hypotheses are: 1) pre-existing, persistent characteristics of schools that affect 
outcomes; and 2) changes in general education policy that might cause changes in outcomes across all 
schools. For example, it is likely that there are preexisting differences between the ELT and non-ELT 
schools (e.g., ELT funding was prioritized to schools serving disadvantaged students) that could 
reasonably account for any observed differences on study outcomes. Similarly, it could be posited that 
changes in federal, state, and district education direction could affect outcomes across some or all schools. 

This study, however, is designed to use a group of comparison schools — matched on key observable 
characteristics as well as pre-program data, when available — to control for many of such potential 
alternative hypotheses. These analyses represent some of the strongest non-experimental methods 
available to get closer to a true estimate of the effect of ELT on teacher and student outcomes (Cook, 
Shadish, and Wong, 2008; Dehejia and Wahba, 1999; Hotz, Imbens, and Klerman, 2006; Morgan and 
Harding, 2006; Morgan and Winship, 2007; Shadish, Cook, and Campbell, 2002; and Stuart, 2010). 

In addition, when longitudinal data are available, the study models use school fixed effects to address the 
first category of alternative hypotheses mentioned above: pre-existing, persistent characteristics of 
schools that affect outcomes. For example, the motivation of parents of students in a school presumably 
affects pre- as well as post-ELT student achievement. Including school fixed effects in the models allows 
the estimation of the effect of ELT over and above what might be expected given the effects of these 
stable school characteristics. The inclusion of year fixed effects does not require establishing a particular 
trend line (or curve); rather, any systematic trends will be captured by including these variables in the 
model. 19 This approach does assume, however, that the pre-ELT trends are the same in ELT and matched 
comparison schools; this assumption is tested in a series of specification tests presented in Appendix F. 

The second category of alternative hypotheses, secular changes in a given academic year expected to 
change outcomes across all schools, are addressed by the inclusion of the matched comparison schools 
and year fixed effects estimated across both ELT and matched comparison schools. To the extent that any 
policy or program implemented in all of the schools affects the matched comparison and ELT schools 
similarly, the year fixed effects will control for their effects, allowing the estimating of the effect of ELT 
over and above the effects of the other factors. 

Models with longitudinal data (and school and year fixed effects) model the implementation of ELT as an 
“interruption” in what would otherwise be assumed to be somewhat stable levels of a particular outcome 
as interest and are thus referred to as interrupted time series (ITS) with a comparison group. While ITS 
models represent a rigorous quasi-experimental design, it should be noted that even after controlling for 
school and year fixed effects, there might still be time-varying, school-specific, unmeasured variables that 


19 In the first two reports prepared for this study, the effect of ELT was estimated by modeling trends in outcomes 
during the years prior to ELT and then projecting this trend into the post-ELT years to estimate the expected 
outcomes in the absence of ELT. This approach, sometimes called “short interrupted time series analysis,” 
provides a strong quasi-experimental design, and it requires at least two assumptions: 1) that achievement trends 
in the pre-ELT years are correctly modeled and 2) that this trend is stable enough to persist unchanged in the 
post-ELT years. Although these assumptions are quite reasonable in many circumstances, it is possible to model 
year-to-year variation in outcomes without making these assumptions. 
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could be related to study outcomes that would not be addressed by controlling for pre-ELT and matched 
comparison school outcomes. These omitted variables could still introduce bias into the estimated effects. 
It is important to note, however, that in combination, the study’s design, use of appropriate statistical 
controls, longitudinal data, and school and year fixed effects yield impact estimates that provide both 
robust and credible evidence of the effects of ELT. 

Our approach to estimating the effects of ELT are described in greater detail later in this chapter. 


Study Sample 

ELT Schools 

To receive grant funds, districts on behalf of designated schools 20 submitted a grant application to ESE 
and were considered as part of a competitive award process. ESE used multiple selection criteria to select 
ELT schools, including: the strength of the application and implementation plans; potential for district- 
wide impact; and the student population served. 21 

Exhibit 2.2 shows the number of schools participating in ELT in each year of the initiative through 2010- 
1 1. At the height of the initiative, 26 schools in 12 districts in Massachusetts participated in ELT. In the 
fifth year of the initiative, 2010-11, 19 schools across 9 districts in Massachusetts participated in ELT. 


20 ELT funds were awarded to designated schools through grants to districts. Grant applications were submitted in 
some cases due to the school’s initiative and in some cases due to the school and district’s initiative. 

21 Funding preference was given to districts with a high percentage of underperforming students on the MCAS 
and/or high poverty rates. 
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Exhibit 2.2: Number of Participating ELT Schools in Each District, by School Year 


District 

2006-07 

(N=10) 

2007-08 

(N=18) 

2008-09 

(N=26) 

2009-10 

(N=22) 

2010-11 

(N=19) 

Boston 

3 

4' 

4' 

4' 

4' 

Cambridge 

2 

2 

2 

2 

2 

Chelsea 



1 

1 

1 

Chicopee 


1 

1 

1 

0 9 

Fall River 

3 

3 a 

3 b 

3 

3 

Fitchburg 


1 

2 

1 c 

1 

Framingham 



2 

1 d 

0 h 

Greenfield 


2 

2 

2 

2 

Malden 

1 

2 

2 

2 

2 

Revere 



2 

2 

2 

Southbridge 



2 

0 e 

0 

Worcester 

1 

3 

3 

3 

2' 


EXHIBIT READS: There were three ELT schools in the Boston Public School district during the 2006-07 school year and 
four ELT schools during the 2007-08, 2008-09, 2009-10, and 2010-11 school years. 

Notes: 

a One ELT school closed due to restructuring, another ELT school merged with a non-ELT school to become a new ELT 
school, and one new ELT school joined the initiative in Fall River in 2007-08. 

b One ELT school in Fall River was again merged, this time with a number of non-ELT schools, to become a new ELT school 
in 2008-09. 

I The two ELT schools in Fitchburg were merged in 2009-10. 

II Participation in the ELT initiative for one school in Framingham was voted down by the teachers’ union in 2009-10. 

e The School Committee in Southbridge voted against continued participation of both ELT schools in that district in 2009-10. 

1 Four schools in Boston received ELT grants. In 2007-08 and 2008-09 all four schools were part of the analytic sample. After 
discussion with ESE, in 2009-10 and 2010-11 only three were part of the analytic sample; a decision was made to exclude one 
school from all aspects of the evaluation because of its unique structure as the only high school implementing ELT and 
because there were no existing (and therefore plausible) comparison schools either within or outside of the district given the 
school’s unique focus as an arts academy. 

E Participation in the ELT initiative for the one school in Chicopee was voted down by the teachers’ union. 
h Participation in the ELT initiative for the remaining school in Framingham was voted down by the teachers’ union. 

1 ELT funding for one ELT school in Worcester was not renewed by ESE in 2009-10 due to underperformance and because its 
Level 4 status made it eligible for Federal SIG funding to aid with school turnaround. 

Source: MA ESE website (http://www.doe.mass.edu/redesign/elt/). 


The implementation and non-academic outcomes analyses include 18 of the 19 schools participating in 
ELT in 2010-11; similar to previous years, one school was again excluded from all aspects of the 
evaluation because of its unique structure. Analyses examining instructional time rely on principal survey 
data collected from 17 of the 18 ELT schools as one ELT school does not serve students in the grades that 
time data was collected for (5 th and 8 th grade). The analysis of non-academic outcomes using teacher and 
student survey data includes the schools that achieved at least a 70 percent response rate; further detail 
about response rates is provided later in this chapter. 

The Student Information Management System (SIMS) behavioral outcomes and MCAS analyses include 
24 ELT schools; for the purposes of estimating impacts on student outcomes from extant data, schools 
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that received funding to implement ELT were maintained in the analytic sample only for those years 
during which the schools implemented ELT. 22 

Exhibits 2.3 and 2.4 present key characteristics of the 18 schools and 9 districts that implemented ELT 
and were part of the study sample in 2010-1 1. Appendix C includes characteristics of sample schools by 
cohort and ELT status. Each school belongs to a particular cohort according to the first year it 
implemented ELT : schools that began implementing ELT in 2006-07 are part of Cohort 1 , schools that 
began implementing ELT in 2007-08 are part of Cohort 2, and schools that began implementing ELT in 
2008-09 are part of Cohort 3. Thus, as of the 2010-11 academic year, schools in Cohorts 1, 2, and 3 have 
been implementing ELT for five, four, and three years, respectively. Exhibit 2.3 presents school 
characteristics by cohort; Exhibit 2.4 presents district information overall given the small number of 
districts (9) in the sample. 

As Exhibit 2.3 indicates, the ELT schools in the study sample included six elementary schools, eight 
middle schools, and four schools that served kindergarten through eighth grade. Elementary and middle 
schools varied in the specific grades served. Two-thirds of ELT schools (12 of 18) served 50 percent or 
more minority populations, and all schools served at least 50 percent low-income students. In just over 
one-third of the schools, more than 20 percent of the population was of Limited English Proficiency 
(LEP). In 2010-11, 16 schools in the ELT sample were identified for “improvement, corrective action, or 
restructuring” based on English Language Arts (ELA) performance, and 17 schools were identified for 
such action on the basis of math performance. 

Exhibit 2.4 suggests that characteristics of the districts where ELT schools are located are reflective of the 
school-level characteristics highlighted in Exhibit 2.3. The majority of the districts served at least 50 
percent minority students (7 of 9) and at least 50 percent low-income students (8 of 9). Further, in about a 
quarter of the districts, 20 percent or more of the students served had limited English proficiency. Finally, 
note that none of the districts met Adequate Yearly Progress (AYP) in 2009-10. All were identified for 
“improvement, corrective action, or restructuring” for English Language Arts (ELA) performance, and all 
but one were identified for such action on the basis of math performance. 


The number of schools included in analyses by implementation year therefore varies, as six schools that had 
implemented ELT for a portion of the study period subsequently discontinued ELT and were removed from the 
study sample after exiting. For example, if a school implemented ELT for two years, it is included in the sample 
when estimating impacts after one or two years of implementation, but is not included in calculating impacts 
after three and four years of implementation. Given the considerable funds necessary to implement ELT, the 
study assumes that schools that have discontinued the initiative no longer extend the length of their school days. 
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Exhibit 2.3: Current Characteristics of ELT Schools, by Cohort, as of 2010-11 



ELT Schools 

Cohort 1 

Cohort 2 

Cohort 3 

Overall 

Grade Span 


Elementary school 

1 

3 

2 

6 

K-8 school 

3 

1 

0 

4 

Middle school 

4 

1 

3 

8 

School Location 


City 

7 

1 

1 

9 

Suburb 

1 

1 

4 

6 

Town 

0 

2 

0 

2 

Rural 

0 

1 

0 

1 

School Size 


600 students or more 

3 

2 

2 

7 

400-599 students 

3 

2 

3 

8 

200-399 students 

2 

0 

0 

2 

Fewer than 200 students 

0 

1 

0 

1 

Low Income Student Population 


75 percent or more 

4 

1 

4 

9 

50-74 percent 

4 

4 

1 

9 

Less than 50 percent 

0 

0 

0 

0 

Minority Student Population 


75 percent or more 

6 

0 

1 

7 

50-74 percent 

1 

2 

2 

5 

25-49 percent 

1 

1 

2 

4 

Less than 25 percent 

0 

2 

0 

2 

SPED Student Population 


20 percent or more 

4 

2 

1 

7 

10-19 percent 

4 

3 

3 

10 

Less than 1 0 percent 

0 

0 

1 

1 

LEP Student Population 


20 percent or more 

5 

1 

1 

7 

10-19 percent 

0 

1 

2 

3 

Less than 1 0 percent 

3 

3 

2 

8 


Met Aggregate Adequate Yearly Progress (AYP) in 2011 


English language arts 

2 

3 

2 

7 

Math 

1 

1 

0 

2 


ELA Accountability Status in 2011 


No status (AYP met for previous two years) 

1 

1 

0 

2 

Identified for improvement, corrective action, or 
restructuring 

7 

4 

5 

16 


Math Accountability Status in 2011 


No status (AYP met for previous two years) 

0 

1 

0 

1 

Identified for improvement, corrective action, or 
restructuring 

8 

4 

5 

17 


EXHIBIT READS: As of fall 201 1, 6 of the 18 ELT schools were elementary schools, 4 were K-8 schools, and 8 were 
middle schools. 


Source: MA ESE website (http://profiIes.doe.mass.edu/). 
Sample: 18 ELT schools in the study in 2010-11. 
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Exhibit 2.4: Characteristics of ELT Districts, 2010-11 



Number of ELT 
Districts 
(N=9) 

District Size 

10,000 students or more 

2 

5,000-9,999 students 

5 

Fewer than 5,000 students 

2 

Low Income Student Population 

75 percent or more 

2 

50-74 percent 

6 

Less than 50 percent 

1 

Minority Student Population 

75 percent or more 

2 

50-74 percent 

5 

25-49 percent 

1 

Less than 25 percent 

1 

SPED Student Population 

20 percent or more 

4 

10-19 percent 

5 

Less than 1 0 percent 

0 

LEP Student Population 

20 percent or more 

2 

10-19 percent 

4 

Less than 1 0 percent 

3 

Met Aggregate Adequate Yearly Progress (AYP) in 2009-10 

English Language Arts 

0 

Math 

1 

ELA Accountability Status in 2011 

No status (AYP met for previous two years) 

0 

Identified for improvement, corrective action, or restructuring 

9 

Math Accountability Status in 2011 

No status (AYP met for previous two years) 

1 

Identified for improvement, corrective action, or restructuring 

8 


EXHIBIT READS: As of fall 2011, of the 9 ELT districts, 2 had 10,000 students or more, 5 had 5,000 to 9,999 students, and 
2 had fewer than 5,000 students. 

Source: District -level data tables downloaded from the MA ESE website (http://profiles.doe.mass.edu/). 

Sample: 9 ELT school districts in 2010-1 1. 


Matched Comparison Schools 

As described above, it could be the case that general changes in federal or state education policy or other 
district- or school-based factors could cause improved outcomes across all schools, not just those 
implementing ELT. To account for this possibility, the study controls for external policy changes by 
including a set of carefully selected schools that are not implementing ELT, since one would expect any 
external shifts to change achievement equally in both ELT and matched comparison schools. Once these 
effects have been removed, one can examine the changes in ELT schools over and above that in their 
matched pair schools and assume that any significant changes were caused by participation in ELT. 
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The assumption that external changes will affect ELT and matched comparison schools equally is based 
on another assumption: that the study’s ELT and matched comparison schools were similar prior to the 
onset of ELT. As such, potential comparison schools were selected to match the ELT schools on as many 
measurable characteristics as possible. 

Non-ELT schools were matched to ELT schools within district 23 and grade span (e.g., K-8, 6— 8) 24 based 
on the following prioritization of matching variables identified in collaboration with ESE staff (see 
Appendix B for a more complete description of these variables): 

Tier 1: Highest priority matching variables 

- ELA Composite Performance Index (CPI) 

- Math Composite Performance Index (CPI) 

- Aggregate ELA Adequate Yearly Progress (AYP) 

- Aggregate Math Adequate Yearly Progress (AYP) 

Tier 2: High priority matching variables 

- ELA Accountability Status 

- Math Accountability Status 

Tier 3: Medium priority matching variables 

- Student enrollment 

- Percent minority (i.e., percent non- White) 

- Percent limited English proficiency (LEP) 

- Percent low-income 

- Percent special education 

Tier 4: Lower priority matching variables 

- Percent male 

- Percent of teachers in core academic subjects who are highly qualified 


23 Within-district matches for two ELT schools could not be found due to the size of the districts. The matched 
comparison schools for these two schools were drawn from demographic ally similar districts in similar regions 
of Massachusetts that also had ELT schools in their districts. 

24 One ELT school has two comparison schools. The original comparison school that was recruited for this ELT 
school only served students in one grade level of interest. As such, a second comparison school was recruited in 
a subsequent year of the study to provide data about both grade levels of interest (5 th and 8 th ). The original 
comparison school was kept in the sample to maintain consistency. 
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Two potential matches were identified for each ELT school. The study team also contacted district 
superintendents to obtain qualitative information to help determine which of the two potential non-ELT 
comparison schools provided a better match. Information about the tenure of the school’s leadership, 
district context, and demographics of the school’s neighborhood allowed the study team to select matched 
comparison schools based on critical quantitative and qualitative factors. 

Again, the goal of the matching process was to create two groups of schools that were very similar prior 
to the implementation of ELT. Similarity was assessed by comparing the two groups, within cohort, in the 
year just before implementation of ELT began on all characteristics that are measured on a continuous 
scale. Because focus is on similarity prior to ELT implementation, all ELT schools (and their matched 
comparison schools) that were ever in the study sample are included in the analysis. The difference 
between the groups on each characteristic was converted into a standardized effect size to assess whether 
the differences might affect the analysis. Recent methodological research suggests that differences larger 
than .25 standard deviations are considered meaningful and therefore require adjustment in the statistical 
models (Ho et al., 2007). 

The results of these analyses are summarized in Exhibit 2.5. Three of the nine comparisons revealed 
differences larger than 0.25 standard deviations (as indicated by A ). In the year prior to ELT 
implementation, matched comparison schools were larger on average, serving about 100 more total 
students. ELT schools, relative to matched comparison schools, were serving an 1 1 percentage point 
greater proportion of students from low income families. One other difference exceeded the 0.25 standard 
deviation cutoff, but the magnitude may be less practically meaningful. On average, matched comparison 
schools had a smaller proportion of core academic teachers designated as highly qualified, although the 
magnitude of this difference was only 3 percentage points. Such a small percentage point difference 
corresponds to considerably large effect sizes as this measure does not vary much across the schools, 
yielding small standard deviations that produced a large effect size. 

These results indicate that while the matched comparison schools may represent the best available 
matches, they differ in some important ways from their ELT counterparts. As a result, where possible, the 
analysis models (described in more detail later in the chapter) used to estimate ELT impacts include 
indicators for each school (called “school fixed effects”) to control for observable and unobservable, time 
invariant, pre-program differences between ELT and matched comparison schools (see Appendix B for 
additional details). 
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Exhibit 2.5: Comparison of School-Level Characteristics of ELT Schools and Matched 


Comparison (MC) Schools in the Final Year Prior to ELT Implementation 



Final Pre-ELT Year 





Difference 


Actual 

Actual 

(in effect size 

Characteristic 

ELT Mean 

MC Mean 

units) 

Student Enrollment 

462 

558 

-0.47 A 

ELA Composite Performance Index 

73 

75 

-0.22 

Math Composite Performance Index 

60 

63 

-0.19 

Percent Minority 

56% 

51% 

0.11 

Percent Limited English Proficient 

12% 

13% 

-0.07 

Percent low income 

72% 

61% 

0.29 A 

Percent special education 

17% 

19% 

-0.10 

Percent male 

52% 

52% 

-0.01 

Percent of teachers in core academic subjects who 
are highly qualified 

97% 

94% 

0.32 A 


EXHIBIT READS: In the year prior to the implementation of ELT, the average enrollment was 462 in ELT schools and 558 
in matched comparison schools. The difference corresponds to an effect size of 0.47 standard deviations. 

Notes: Means shown are averages across schools within each respective group. 

A effect size of at least 0.25 standard deviations 
Source: MA ESE website ( httv .-//profiles. doe. mass. edit/ ). 

Sample: Full sample of 49 schools in the outcomes study, including 24 ELT and 25 matched comparison schools in 2010-11. 


Data Collection Measures 

This section describes the data collection measures used in 2010-1 1 for each study component. As 
summarized in Exhibit 2.6, the implementation component of the study draws on data from interviews 
conducted with a variety of stakeholders in the ELT initiative — representatives from ESE, Mass 2020, 
and Focus on Results, district administrators, and school principals — as well as from surveys 
administered to teachers and students. The outcomes component of the study also uses data from these 
surveys as well as extant data provided by the state; additionally, the outcomes component of the study 
uses some select data from interviews with principals, specifically school-specific time allocation to 
various instructional activities. Each data collection measure is described in more detail below. 
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Exhibit 2.6: Data Collected in 2010-11, By Source and Study Component 



Implementation 

Component 

Outcomes 

Component 

Interviews 



ESE Representatives 

V 


Mass 2020 Representatives 

V 


Focus on Results Representatives 

V 


ELT District Administrators 

V 


ELT Principals 

V 

V 

Matched Comparison (MC) Principals 

V 

V 

Surveys 



ELT Teachers 

V 

V 

ELT Students 

V 

V 

MC Teachers 


V 

MC Students 


V 

MCAS Scores 



ELT Students 


V 

MC Students 


V 

Other Extant Data 


V 


Interviews and Focus Groups 

Abt developed semi-structured interview protocols to gather information about key aspects of ELT 
implementation; data gathered about time allocation was of particular' focus in this year’s analyses. The 
interview protocols are in Volume II of this report. 

Teacher Surveys 

During spring 2011, as in prior years, teachers in both ELT and comparison schools responded to surveys 
organized into six topic areas: school environment; time use; enrichment and academic support; school- 
wide academic focus; individual and collaborative planning; and professional development. Teacher 
surveys were identical for ELT and matched comparison schools, except that within each section of the 
survey, ELT-specific questions were included, as appropriate, for teachers in ELT schools (such questions 
were not included on surveys for comparison schools’ teachers). Copies of the teacher surveys are in 
Volume II. 

Student Surveys 

During spring 201 1, as in prior years, 5 th and 8 th grade students were surveyed about their perceptions of 
and attitudes towards school, relationships with teachers, and participation in extracurricular activities. 
The student survey is in Volume II. 

Publicly Available School-Level Datasets 

Publicly available school-level datasets for both ELT schools and matched comparison schools were 
downloaded from the ESE website (http://profiles.doe.mass.edu/), for each school year available 
beginning as early as 2001-02 through 2010-11. The datasets include all of the variables used in the 
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matching process described above (school-level academic achievement indicators, school-level student 
demographic characteristics and special population designations, school enrollment, and percent of core 
academic teachers identified as highly qualified). Data prior to the implementation of ELT were used to 
select potential matched comparison schools, and the final sample of comparison schools was selected 
using the process outlined in the previous section. Data both prior to and after the implementation of ELT 
were used to examine the effects of ELT on these school-level outcomes and as covariates in models 
estimating the effects on student and teacher survey responses. 

Student-Level Data: Achievement, Attendance, Suspensions and Truancy 

ESE provided longitudinal, student-level MCAS data for both the ELT and the matched comparison 
schools. These datasets include student-level achievement (proficiency levels, raw scores, and scaled 
scores) on the Reading/English Language Arts, Math, and Science MCAS exams from 2001-02 through 
2010-11 as well as student-level characteristics (e.g., special education status). 

Data provided from ESE also included information from the end-of-year Student Information 
Management System (SIMS) files 25 , such as student-level demographic variables and behavior variables 
including attendance rate, truancy rate, in-school suspension rate, and out-of-school suspension rate 
(percent of school days). Data are available for all students through the 2010-11 school year. Attendance 
data are available beginning in 2001-02, whereas data for the other variables are available beginning in 
2003-04. 


Data Collection Administration 

Interviews 

As shown in Exhibit 2.7, a variety of stakeholder groups were interviewed in the study’s fifth year. 
Interviews were conducted with principals of all 1 8 ELT schools that were part of the study sample as 
well as with all 19 of their matched comparison schools included in the study this year. The coordination 
and scheduling of principal interviews as well as the provision of school schedules used to compute time 
allocations to instructional activities were facilitated by an internal study liaison at each ELT and matched 
comparison school. Principal interviews were conducted in person; they were generally an hour and thirty 
minutes long and were followed by a brief tour of the school building. The interviews occurred between 
March and June of 201 1 . 

The study team also interviewed district leadership representatives from eight of nine districts with ELT 
schools by telephone as well as representatives from ESE, Mass 2020, and Focus on Results. Interviews 
ranged from 30 minutes to 60 minutes (for district and state and technical assistance personnel, 
respectively), and were conducted during the fall 2011. In total, 59 respondents participated in interviews. 


25 SIMS data are collected three times each school year: October 1, March 1, and at the end of the school year. 

For more information about the types of data available in the SIMS database, see the Massachusetts Department 
of Elementary and Secondary Education's website: 
http://www.doe.mass.edu/infoservices/data/sims/DataHandbook.pdf 


pg. 28 | Chapter 2. Study Design 


Abt Associates Inc. 



ELT Evaluation Year 5 Final Report 


Exhibit 2.7: Number of Interview Respondents, 2010-1 1 



Number of 
Interviews 

Number of 
Respondents 

ESE Representatives 

3 

5 

Mass 2020 Representatives 

3 

3 

Focus on Results Representatives 

1 

1 

ELT District Administrators a 

8 

9 

ELT Principals b 

19 

21 

MC Principals c 

19 

20 

Total 

53 

59 


EXHIBIT READS: In 2010-11, three interviews were conducted with representatives from ESE, and there were a total of 
five respondents from ESE. 

Notes: 

J The study team conducted interviews with eight of nine administrators in districts with ELT schools. Additionally, one 
district administrator invited a colleague to participate in the interview. 

b In the 2010-11 school year, the study conducted interviews with 19 principals in 18 schools (one K-8 school had two 
principals, one each for the elementary and middle grades). Additionally, two principals invited a member of their 
administrative team to also provide input during the interview. 

1 The study conducted interviews with all 19 matched comparison school principals, one of whom also invited a member of the 
administrative team to participate in the interview. 


Prior to adult participation in one-on-one (or small group) interviews, informational materials, including 
consent forms, were distributed to participants, and interviews began only after participants provided 
verbal consent. Before data collection, Abt staff participated in a training session to ensure that they: 1) 
shared a common understanding of the study background, objectives, design, and schedule; 2) understood 
the purpose of the interviews; 3) were fluent in administering each question in the interview protocol and 
associated probe questions; 4) understood the logistics of scheduling and conducting interviews and the 
calendar for data collection; and 5) reviewed data security, data recording, and submission procedures. 
Most interviews were facilitated by two members of the Abt study team; generally, one person led the 
discussion while the other took notes. When interviews were conducted by a single individual, the 
discussions were tape recorded and used for reference to ensure that the notes accurately reflected 
discussions; tapes were not systematically transcribed. All interview responses were entered into a 
standardized Microsoft Access database. 

Surveys 

All staff members who provided instruction to students and all 5 th and 8 th grade students in the 
participating schools were asked to complete brief paper surveys between May and June of 201 1. Survey 
administration activities in 2010-11 were standardized, involved systematic tracking, and offered 
participants a variety of incentives. 

Teacher Surveys 

All staff members who were responsible for providing instruction to students during academic classes, 
specials, and/or enrichment activities — regardless of individual staff members’ participation in the 
expanded day schedule — were asked to complete the teacher survey. To standardize the definition of what 
was meant by “instructional staff’ across schools, the study team obtained the most recent Educational 
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Personnel Information Management System (EPIMS) data available from ESE. Job classification and 
work assignment codes were used to create individual rosters for each participating school’s instructional 
staff. 

A designated study liaison, usually the ELT coordinator or principal, facilitated the data collection at each 
ELT and matched comparison school. The study team provided the liaison with instructions about survey 
administration and deadlines. School liaisons were asked to review rosters and indicate changes in 
personnel. Each instructional staff member was assigned a unique staff ID number used to track response 
rates. 

The schools liaisons determined how best to distribute and collect teacher surveys; some schools asked 
teachers to complete surveys during regularly scheduled faculty meetings, while other school liaisons 
distributed surveys and asked instructional staff to return surveys within a specified time frame. 

The surveys themselves included an information sheet that communicated details about the study, the 
surveys, and the confidential and voluntary nature of the surveys; the completion of the survey itself 
served as an indication that the teacher consented to do so. The surveys took teachers approximately 20- 
30 minutes to complete. All completed surveys were sealed in envelopes to ensure respondent 
confidentiality. 

Student Surveys 

All 5 th and 8 th grade students in study schools were asked to complete the student survey. The study team 
created school-specific rosters of all 5 th and 8 th grade students by homeroom, and homeroom teachers 
were asked to review rosters and indicate changes in class enrollment as well as students’ eligibility to 
participate in the study (e.g., students with disabilities or limited English language proficiency were 
exempt from participation if school staff felt it was inappropriate for them to answer the survey 
questions). 

As with the teacher survey, a designated study liaison, usually the ELT coordinator or school principal, 
facilitated data collection at each school. Liaisons generally relied on students’ homeroom teachers to 
administer and collect the student surveys, and to indicate, on their respective classroom rosters, which 
students had completed surveys. The surveys took students approximately 15-20 minutes to complete. All 
surveys and rosters were sealed in envelopes to ensure respondent confidentiality. 

Strategies for Encouraging High Response Rates 

High response rates ensure the representativeness of survey responses within a school. As described in 
greater detail below, a response rate under 70 percent for a given respondent group within a school was 
considered not sufficiently representative and resulted in the particular respondent group’s exclusion from 
study analyses. An incentive structure was built into the survey administration process to encourage high 
response rates. As in past years, all participating schools received a school-specific report describing the 
most recent school year’s survey results for that school. Liaisons at both ELT and matched comparison 
schools received an honorarium for assistance coordinating the survey administration. Matched 
comparison schools received an additional honorarium for their participation. Each instructional staff 
member who completed a survey received a small gift card. In 2010-11, as in 2009-10, the study team 
provided additional incentives at the classroom and school levels. All homeroom teachers who returned 
completed surveys from 75 percent or more of eligible students were offered a small stipend for 
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educational materials. Each school that returned completed surveys from 75 percent or more of eligible 
teacher and students was entered into a lottery; the selected winner received $2,000. 26 

The Abt study team was in regular contact with school liaisons during the survey data collection effort to 
monitor their progress and troubleshoot solutions to any barriers that arose. The unique staff ID numbers 
as well as the student rosters also allowed study staff to track each school’ s survey response rates and to 
engage in follow-up efforts to increase response rates. ESE also sent a memo to participating schools 
indicating the importance of the study. 

Data Entry 

Upon receipt of the completed surveys, staff at Abt Associates conducted initial processing to track 
survey response rates and prepare surveys for data entry. Data entry itself was conducted by Public 
Consulting Group. Public Consulting Group then provided the survey datasets to Abt Associates for 
analyses. 

Survey Response Rates for Students and Teacher: Implications for Analysis and Interpretation of 
Findings 

Exhibit 2.8 summarizes survey response rates by respondent group (teachers and students). Schools in 
which the response rate for a particular respondent group was below 70 percent were excluded from 
analyses of outcomes for that group, as were their matched pairs. The survey sample included 1,808 
teachers (918 teachers from ELT schools and 890 teachers from matched comparison schools) and 4,840 
students (2,300 students from ELT schools and 2,540 students from matched comparison schools). 


26 Two separate lotteries were held, one among ELT schools and one among matched comparison schools that met 
survey response rate thresholds. 
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Exhibit 2.8: Survey Response Rates, 201 0-1 1 



Teacher Surveys 

Student Surveys 

Overall 

Min 

Max 

Overall 

Min 

Max 

ELT 

Overall 

(N=1 8)* 

89% 

76% 

1 00% 

96% 

85% 

1 00% 

Cohort 1 

(N=8) 

88 

76 

100 

94 

85 

100 

Cohort 2 

(N=5)* 

91 

79 

100 

97 

94 

100 

Cohort 3 

(N=5) 

91 

84 

96 

97 

96 

99 

Elementary 

(N=1 0)* 

90 

76 

100 

99 

96 

100 

Middle 

(N=8) 

89 

79 

96 

94 

85 

98 

MC 







Overall 

(N=1 9)* 

79 

64 

97 

88 

70 

1 00% 

Cohort 1 

(N=8) 

79 

68 

97 

85 

70 

98 

Cohort 2 

(N=5)* 

85 

77 

97 

95 

92 

100 

Cohort 3 

(N=6) 

75 

64 

81 

87 

75 

100 

Elementary 

(N=1 1)* 

77 

64 

97 

97 

85 

100 

Middle 

(N=8) 

81 

72 

91 

83 

70 

94 


EXHIBIT READS: Eighteen total ELT schools administered teacher surveys. These schools achieved an overall teacher 
survey response rate of 89%. 

*Note: One ELT and one matched comparison school (Cohort 2 elementary schools) did not serve students in the target grade 
levels, 5 lh and 8 th . and therefore did not administer student surveys. 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, and Abt Associates' Surveys ofMA ELT and 
Matched Comparison School Students, Spring 2011. 


Teacher Response Rates 

Teachers from 37 schools (18 ELT and 19 matched comparison schools) were surveyed. The response 
rates across the schools ranged from 64 to 100 percent. Eighteen ELT schools and 17 matched 
comparison schools achieved response rates of at least 70 percent. 27 The subsample of schools included in 
the analysis differs slightly from those schools excluded from the analysis on a number of measurable 
characteristics (see Exhibit 2.9). 


11 The two matched comparison schools that were excluded from analyses based on response rate represent one 
Cohort 1 and one Cohort 3 school, and both are elementary schools. 
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Exhibit 2.9: Comparison of School-Level Characteristics of Schools Included and Excluded 
from the Analysis of Teacher Survey Responses, 2010-11 



ELT Schools 

MC Schools 

Characteristic 

Schools 
Included 
( n = 16) 

Schools 
Excluded 
(n = 2) 

Schools 
Included 
( n = 17) 

Schools 
Excluded 
(n = 2) 

Student enrollment 

587 

523 

571 

619 

ELA CPI 

79 

75 

80 

75 

Math CPI 

70 

70 

69 

71 

Percent minority 

64 

59 

61 

40 

Percent Limited English Proficient 

19 

7 

17 

4 

Percent low income 

77 

75 

70 

60 

Percent special education 

19 

16 

21 

23 

Percent male 

51 

51 

53 

53 

Percent of teachers in core academic subjects who are 
highly qualified 

97 

94 

97 

98 


EXHIBIT READS: The average enrollment was 587 in ELT schools included in the analysis of teacher survey responses 
and 523 in schools excluded from the analysis. The average enrollment was 571 in matched comparison schools included 
in the analysis of teacher survey responses and 619 in matched comparison schools excluded from the analysis. 

Notes: Means shown are averages of the school-level means within each respective group. 

Source: School-level data from MA ESE website. 

Sample: Thirty-seven schools across three cohorts. 


Student Response Rates 

Students from 35 schools (17 ELT and 18 matched comparison schools) were surveyed. 28 The average 
response rates across the schools ranged from 70 to 100 percent. Because all of the ELT and matched 
comparison schools achieved response rates of at least 70 percent, no schools were excluded from student 
survey analyses. 


Approach to Analyzing ELT Implementation Data 

All interview data were transcribed by study staff, and data were synthesized and entered into a 
standardized Access/relational database. Data were then cleaned to ensure that study staff accurately and 
consistently interpreted various study-related terminology, particularly concerning definitions of different 
instructional activities and how to calculate allocations of time across the school day. After interview data 
were reviewed, data in the master database were categorized by topic to produce topic-specific reports. 
Responses were then summarized both qualitatively and quantitatively for analysis and reporting. It is 
important to note that the perceptions of individuals presented in this report do not necessarily reflect the 
perceptions of all individuals within a given school or organization. 

The study team reviewed interview and survey data about implementation in regular analytic meetings as 
well as whole-team debriefing meetings. The purpose of the whole-team meetings was to discuss 
emerging themes, share interviewing strategies, as well as to discuss any improvements to the logistics of 

28 One ELT and one matched comparison school did not serve students in the target grade levels, 5 th and 8 th , for 
the survey administration. 
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the visits and study protocols. The analytic meetings focused on discussion of patterns, developing 
analytic strategies for presentation of descriptive data, and revision and application of the implementation 
index (described in more detail in Chapter 3). 


Approach to Estimating the Effects of ELT on Non-Academic and Academic 
Outcomes 

As described above, addressing the key research questions described above for the outcomes study 
involves estimating the difference between outcomes for teachers and students in ELT and matched 
comparison schools so as to control for potential alternative hypotheses. The specific approaches used to 
model the effects of ELT on various outcomes for schools, students, and teachers are described below. 

The section begins by discussing the approaches used to assess the non-academic outcomes presented in 
Chapter 5. These outcomes, which rely primarily on student and teacher data, are generally cross- 
sectional, but most control for school, student, and teacher characteristics, and adjust for clustering at the 
school and, where appropriate, classroom level (see Appendix B for formal model specifications). The 
section continues by discussing the approaches used to model the outcomes assessed in Chapter 6, 
including the relationship between ELT and student mobility and the effects of ELT on school 
characteristics and MCAS scores. Because longitudinal data are available to assess the effects of ELT on 
school characteristics and MCAS scores, these short interrupted time series with a comparison group are 
the most rigorous models used in the study. The inclusion of school and year fixed effects allows for a 
more robust model specification than what is feasible when assessing the bulk of the non-academic 
outcomes. This section also discusses how analyses are conducted across cohorts by implementation year. 
The section concludes with a description of how to interpret the results of these analyses. In addition, to 
facilitate the interpretation of key findings later in the report, each outcome chapter (Chapters 5 and 6) 
includes a brief review of salient features of the approach. 

As noted earlier, while attempts have been made to use the most rigorous quasi-experimental methods 
available, there might still be time-varying, school-specific, unmeasured variables that could be related to 
study outcomes that would not be addressed by controlling for pre-ELT and matched comparison school 
outcomes. These omitted variables could still introduce bias into the estimated effects. 

Modeling the Effects of ELT on Non-Academic Outcomes: Time Use and Student and Teacher 
Survey Outcomes 

Analyses comparing differences between ELT and matched comparison schools in time allocated to 
various instructional activities (e.g., ELA, math) draw on data from principal interviews and regresses 
time on an ELT indicator and dummy variables for each pair of matched schools. The ELT indicator flags 
significant differences in time between ELT and matched comparison schools, while the matched pair 
dummy variables reflect the design of the study, wherein it is assumed that there are unobserved 
characteristics unique to each matched pair. 

To address questions about the effect of ELT on student and teacher survey outcomes, the study uses 
multi-level models wherein students are nested in classrooms, and schools and teacher are clustered in 
schools. Note that because of the cross-sectional nature of the survey data, these models are different from 
what is used for the academic outcomes. Since the study does not have pre-ELT survey data, it is not 
possible to use school or year fixed effects to control for observed and unobserved, persistent 
characteristics of schools or secular year-to-year trends. Consequently, these models do not produce as 
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rigorous evidence as the academic outcome models described below, and the findings should be 
interpreted with considerably more caution. 

Student and teacher survey outcomes are regressed on a school level variable that indicates whether the 
student is in an ELT school, school characteristics, 29 student/teacher demographic variables, 30 and a 
dummy variable for each matched pair of schools. Student models appropriately adjust for clustering of 
students within classrooms and schools, while teacher outcomes adjust for the clustering of teachers 
within schools. 

Prior to analysis, the study team identified the student and teacher survey items that corresponded with 
the ELT outcomes of interest (e.g., student engagement, collaborative leadership). Recognizing that there 
were too many items in each area of interest to present all of them in the main body of the chapter, the 
team used a technique called factor analysis to reduce the number of survey items presented. All survey 
items from teacher or student surveys that addressed a non-academic outcome of interest were grouped 
together in a single domain. The three survey items from the student or teacher surveys that loaded the 
highest for a particular domain were selected to be presented in the chapter. 31 If there were less than three 
survey items per survey type (e.g., three student survey items and/or three teacher survey items) in a 
particular domain, all three items were presented. A comprehensive presentation of all student and teacher 
survey items corresponding to outcomes of interest can be found in Appendix E. 

Finally, while Chapter 5 primarily uses cross-sectional interview and survey data, there are a handful of 
non-academic outcomes (e.g., attendance, suspension rates) that draw on longitudinal SIMS data. As 
such, modeling of these outcomes more closely mirrors the impact models (see discussion below on 
academic outcomes) that include year fixed effects to control for secular year-to-year fluctuations in 
outcomes. In addition, unlike data drawn from other sources (including MCAS data), the SIMS data are 
unique in that students can be linked across multiple school years. As a result, these models also include 
student fixed effects, which control for all of the observable and unobservable persistent characteristics of 
individual students over time. 

Modeling the Effects of ELT on Academic Outcomes: School Characteristics and MCAS Scores 

The staggered implementation of the ELT initiative through multiple cohorts of schools beginning in 
multiple years provides the opportunity to analyze outcome data in two different ways: separately for 
each cohort in each calendar year, or pooled within one of three dimensions (cohort, calendar year, or 
implementation year). While it is possible to examine outcomes for a single cohort within a single 
calendar year, such analyses would have little statistical power to find effects given the small sample size. 


29 The school-level characteristics included in the model measured total enrollment, the percentage of students 
eligible for free and reduced priced lunch, and the percentage of minority students served by the school during 
the 2010-11 school year. 

30 Student demographic variables include gender, and whether or not the student reported speaking a language 
other than English most of the time at home. The teacher-level demographic variables measure the total number 
of years the teacher reported teaching/working in a school, including 2010-11 school year. 

31 One exception to this rule was a domain that included questions about teacher satisfaction for time available for 
instruction in various subjects, in which all subjects were presented as each represented a unique focus (whereas 
items in other domains pertained to one underlying constmct). 
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Pooling increases the sample on which to base estimates, improving statistical power; however, it is then 
no longer possible to examine differences along the dimension across which the data are pooled. Pooling 
therefore improves the study’s capacity to answer some research questions, while hindering its capacity to 
address others. Choosing the method by which to pool the data therefore must be driven by which 
research questions are of the highest priority. 

As of 2010-11, all three cohorts of schools have implemented ELT for at least three years, yielding a 
reasonable sample on which to base estimates of effects of ELT after at least one, two, and three years of 
implementation. In addition, two cohorts of schools have completed four years of implementation and 
therefore can contribute to the estimate of the effect of ELT after four years of implementation. Since 
only one cohort of schools (Cohort 1) has completed five years of implementation, it is premature to 
present these findings in the main body of this report, and, as such, fifth year findings are included in 
Appendices D and E. 

Models that assess the effect of ELT on school- level characteristics (e.g., the extent to which the 
implementation of ELT changes the characteristics of students and teachers in ELT schools) compares 
school-level data for ELT and matched comparison schools both prior to and after the implementation of 
ELT. Using both pre- and matched comparison school data allows the study to control for persistent, 
school-specific factors that could explain any observed and unobserved differences, thus reducing the 
number of plausible alternative explanations for the estimated effects. The model also includes school 
fixed effects, which control for the effects of all of the observed and unobserved persistent characteristics 
of these schools, including those that may differ between ELT and matched comparison schools. Year 
fixed effects control for any variation in outcomes that is unique to a specific calendar year but is 
consistent across schools (both ELT and comparison) in the analytic sample. 

The effect of ELT on school-level outcomes is modeled using five variables: the first indicates whether 
the outcome is from an ELT school when it is in its first year of implementation (and is coded zero 
otherwise), the second whether the outcome is from an ELT school when it is in its second year of 
implementation, the third whether the outcome is from an ELT school in the third year of implementation, 
the fourth whether the outcome is from an ELT school in the fourth year of implementation, and the fifth 
whether the outcome is from an ELT school in the fifth year of implementation. These variables provide 
separate estimates of the effects of ELT across cohorts during their first, second, third, fourth, and fifth 
year of implementation over and above any changes in the outcomes that might be expected given 
outcomes prior to ELT and in schools not implementing ELT 32 . 

The effect of ELT on mobility rates is assessed by comparing average ELT student mobility rates in each 
implementation year to average student mobility rates in matched comparison schools. Differences in 
means are assessed using a two-tailed independent t-test that assumes equal variance between samples. 

To address questions about the effect of ELT on student-level academic outcomes (MCAS scores), the 
study uses an approach almost identical to the one described for estimating ELT impacts on school-level 


32 Note, however, that because only Cohort 1 and 2 schools have outcomes in implementation years four and five, 
estimated impacts in these years are incomplete (i.e., they do not account for the full study sample); thus, one 
cannot make definite conclusions about the relationship between implementation years four and five and 
student outcomes. 
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characteristics. However, because data for these outcomes are on the student-level, models include 
student-level covariates to control for their effects on outcomes and to increase the precision of the 
estimates, 33 in addition to school and year fixed effects variables and variables to indicate what 
implementation year ELT schools were in. Further, the models appropriately account for clustering of 
students in schools using the cluster-robust variance estimator (also known as the “sandwich” standard 
errors; White, 1984 and Liang and Zeger, 1986). 

In addition to the academic outcome models described above, Appendix F includes a number of 
specification and sensitivity models that test whether the results of the student achievement analysis 
would have differed had different analytic decisions been made. The specific scenarios tested included the 
following: 

• Tests using an alternative specification of the model, one that included pre-ELT implementation 
group (ELT versus matched comparison) trends in performance; 

• A test that included all ELT schools in all implementation years, rather than excluding schools 
from the analyses after they exited the sample; and 

• Tests reducing the sample to only five and only three years of pre-ELT implementation data for 
each cohort, using both the primary model and the alternative specification with pre-ELT group 
trends. 

Interpreting the Results 

The goal of these analyses is to estimate the impacts (e.g., MCAS scores for students or perceptions from 
teachers) in ELT schools and to test whether these outcomes are different from what would be expected in 
the absence of ELT, often referred to as the counterfactual. The study can observe the actual outcomes for 
ELT students and teachers, but it is not possible to observe the outcomes for these same individuals 
without ELT. Since it is not possible to observe the outcomes of students and teachers in the absence of 
ELT, the study uses a statistical technique to estimate the counterfactual using data from students and 
teachers in schools that did not implement the program (i.e., the matched comparison schools). Once the 
models have been estimated, the responses are no longer referred to as averages for matched comparison 
schools, since they do not accurately reflect the real outcomes in matched comparison schools. Rather, 
they are regression-adjusted estimates of what would have happened in ELT schools if the initiative had 
not been implemented. Therefore, all findings are described in terms of the actual outcomes measured in 
ELT schools relative to the estimated outcomes in the counterfactual. If differences are observed, there is 
evidence that those differences can be attributed to the effects of the implementation of ELT. 

Limitations 

Despite attempts to control for as many unobserved variables that may affect outcomes as possible, there 
are a number of limitations to the study’s analytic approaches that should be reiterated prior to presenting 
findings. For example, results that rely on student and teacher surveys are limited by a number of 
considerations, including the data structure and response rates. As discussed earlier in the chapter, models 
that assess survey outcomes are cross-sectional and thus do not include the school and year fixed effects 


33 These covariates included gender, minority status, low income status, limited English proficiency status, and 
special education status. 
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that are key to the short ITS design employed in the longitudinal analyses. As a result, these models 
cannot control for any pre-existent, persistent characteristics of schools that affect outcomes or secular 
changes in education policy that may affect all schools. In addition, the representativeness of these results 
is subject to survey response rates, which, while high, may arguably not generalize to all teachers or 
students in a particular school. Similarly, while analyses that use longitudinal data to assess the impacts of 
ELT on school and MCAS outcomes are more rigorous than those used to model survey outcomes, they 
may still omit important unobservable characteristics that affect outcomes. For example, despite the 
inclusion of school fixed effects, it is possible that there are important unobserved school characteristics 
that vary over the study period and thus are not controlled for by the school fixed effects. Likewise, if 
secular changes in a given academic year affect ELT and matched comparison schools differently, then 
the year fixed effects included in the models will not be sufficient to control for these differences. 
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Chapter 3. Implementation Index 


This chapter presents summary information based on an index developed to quantify the level of 
implementation for each school and the initiative as a whole by comparing implementation of core ELT 
components at ELT and matched comparison schools. Chapter 4 analyzes in more detail, by core 
component, the extent of implementation in ELT schools. 


Purpose 

After several years of studying both the implementation and outcomes of the ELT initiative, ESE and Abt 
realized that the information collected thus far was insufficient to understand variation in implementation 
across the ELT schools. To that end, the Abt study team, working collaboratively with ESE and Mass 
2020, developed an index during 2009-10 and used it to analyze data from the 2009-10 school year 
related to quantity of time and time use much more explicitly. This index was modified and restructured 
in 2010-11 to reflect the most current expectations of ELT schools and best capture the data collected 
from schools during this year. 

The index builds on five foundational tenets of the ELT model articulated by Mass 2020 and encouraged 
by ESE, which are described in more detail below. The index uses data from interviews with principals as 
well as from surveys of teachers and students; as such, it provides a snapshot of ELT implementation 
using interviews and surveys at a particular point in time. It should be noted that information about school 
schedules (and time allocation) was collected only for fifth and eighth grade students, and only students in 
those grades completed surveys. 34 

The implementation index attempts to capture information about implementation both at the individual 
school level and for schools overall. Its criteria can be applied to both ELT and comparison schools, an 
important feature given that (1) some districts have begun to expand time in additional schools beyond 
those funded by the state’s ELT initiative, and (2) some of the core expectations of ELT apply across 
multiple school reform initiatives, and consequently are not limited to the ELT initiative. As such, the 
index was used to measure levels of implementation in both ELT and matched comparison schools during 
2010 - 11 . 


Description of Index 

The implementation index (included in Appendix C) includes five criteria based on core expectations for 
ELT implementation by ESE, Mass 2020, and Focus on Results, including: 

1) school-wide academic focus; 

2) core academics; 

3) enrichment activities; 

4) teacher leadership and collaboration; and 

5) school leadership. 

34 The index developed for this study is distinct from tools developed by ESE and Mass 2020 to monitor and 
measure school-level implementation. 
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Each criterion includes four levels, ranging from zero (indicating no or little discernable activity), to 
Level 1 (representing limited evidence that schools have implemented activities related to that criterion), 
to Level 2 (representing modest evidence that schools have implemented activities related to that 
criterion), to Level 3 (representing consistent evidence that schools have implemented activities related to 
that criterion). Each consecutive level assumes that all the conditions outlined in the prior level have been 
met. Three of the criteria described above (core academics, enrichment, and teacher leadership and 
collaboration) are measured by two components, amount of time engaged in the activity and how the time 
is used. 

The index also includes a sixth criterion that is only examined for ELT schools. The sixth criterion 
focuses on the level of support for ELT within schools and districts. The implementation index is 
predicated upon the presence or absence of its structural elements; in most cases, it does not represent a 
measure of the perceived or observed quality of those elements. 

Criterion 1: School-Wide Academic Focus 

The academic focus criterion addresses whether a school-wide academic focus is present and consistently 
identified and disseminated (Level 1), whether the focus is linked to instruction (Level 2), and whether 
the focus is regularly monitored and used to adjust instructional practices (Level 3). 

Criterion 2: Core Academics in Target Grades 

The time component of the core academics criterion addresses whether students in target grades receive at 
least 300 minutes of ELA and at least 250 minutes of math instruction per week (Level 1), at least 350 
minutes of ELA instruction, at least 300 minutes of math instruction, and at least 150 minutes of science 
or social studies instruction per week (Level 2), and at least 200 minutes of both science and social 
studies instruction in addition to Level 2 ELA and math thresholds (Level 3). The “how” component 
addresses teacher support for improvements to academic instruction and student receipt of at least 50 
minutes of dedicated academic support time per week (Level 1), student receipt of at least 100 minutes of 
academic support time per week and whether specific particular' instructional practices are used (Level 2), 
and student receipt of at least 150 minutes of academic support time in addition to a higher threshold for 
the use of particular instructional strategies (Level 3). 

Criterion 3: Enrichment Activities in Target Grades 

The time component of the enrichment criterion addresses whether the school offers at least 45 minutes of 
enrichment activities and 45 minutes of specials per week (Level 1), at least 90 minutes of each per week 
(Level 2), and at least 135 minutes of each (Level 3). The “how” component addresses student access to 
enrichment (Level 1), student choice about and school integration of enrichment (Level 2), and the extent 
to which enrichment activities are influenced by the school’ s academic focus, connected to grade level 
standards and/or curriculum frameworks, and perceived by teachers to be of high quality (Level 3). 

Criterion 4: Teacher Leadership and Collaboration 

The time component of the teacher leadership and collaboration criterion addresses whether the majority 
of teachers attend collaborative planning meetings at least twice monthly for at least 45 minutes (Level 1), 
at least once per week and whether/how frequently collaborative planning time has been supplanted by 
other unrelated activities (Level 2), and more than once per week (Level 3). The “how” component 
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addresses the structure and content of collaborative planning time (Level 1), the culture of collaboration 
(Level 2), and opportunities for teacher leadership in a school (Level 3). 

Criterion 5: School Leadership 

The school leadership criterion addresses teacher perceptions of school leadership direction setting (Level 
1), school leadership involvement in instructional practice and staff development (Level 2), and school 
leadership focus on strengthening school culture (Level 3). 

Criterion 6: ELT Support 

A subset of indicators of ELT implementation applicable only to funded ELT schools comprises the ELT 
support criterion. This criterion addresses the extent to which teachers are supportive of ELT and report 
that school leadership sets high expectations for ELT (Level 1), whether and how the leadership team 
involves others in designing and implementing ELT changes and teacher perceptions of district support 
for ELT implementation (Level 2), and if the majority of teachers report satisfaction with staffing 
provided to cover the entire school day (Level 3). 


Applying the Implementation Index 

The specific thresholds for components of the index were determined using an iterative process, and 
decisions were tempered by expectations of what might be reasonable for schools to achieve after three, 
four, or five years of implementation. Each school received a score for each criterion that ranged from 
zero (insufficient evidence of achieving Level 1) to three (corresponding to Level 3 of implementation); 
criteria with two components (amount of time and how time was used) generated two scores, resulting in 
eight separate scores, and a maximum total of 24 points. Note that a zero does not necessarily mean that a 
school did not implement a particular criterion at all; it simply means that it did not meet the threshold 
necessary for Level 1. Each criterion was weighted equally. The ELT support criterion applied only to 
ELT schools and was not used to calculate the overall score. 

Two comparison schools could not be rated on the six criteria that required teacher survey data (criterion 
1, 2 “how”, 3 “how”, 4, 4 “how”, and 5) because they did not meet the 70 percent or higher teacher 
survey response rate threshold. Two schools (one ELT and one comparison school) could not be rated on 
criterion 2 and 3 about the amount of time allocated to core academics and enrichment because they did 
not serve students in the target grades about which information regarding time allocations was gathered. 
Since these four schools are missing scores on individual criterion, they were not assigned an overall 
implementation index score. 


Key Findings 

Key findings about implementation based on application of the implementation index include: 

• The range of total scores for ELT schools was from 5 to 22, and for comparison schools, the 
range was from 2 to 12 out of a total possible score of 24. 

• The average total score for ELT schools was 1 1.4 and for comparison schools was 6.9. 

• ELT schools, on average, scored higher than comparison schools on six of the eight criteria. 


Abt Associates Inc. 


Chapter 3. Implementation Index | pg. 41 




ELT Evaluation Year 5 Integrated Report 


• Comparison schools scored notably lower on the enrichment-related criteria. 

• In almost all cases, the average score on each criterion across ELT schools was between one and 
two (or between Level 1 and Level 2). In almost all cases, the average score on each criterion 
across comparison schools was between zero and one (or between Level 0 and Level 1). 

• Many comparison schools also appeared to be implementing at least some of the key components 
that are considered core expectations of the ELT initiative. Lor two criteria, the average 
comparison school was higher than the average ELT score. 

The range of total scores for both ELT and comparison schools was large. ELT school scores ranged from 
a total of 5 to 22, and comparison schools ranged from 2 to 12. Six ELT schools scored higher than 12, 
the score of the highest comparison school. Ten ELT and four comparison schools scored higher than the 
mean overall average total score, which was almost 9. The average total score for ELT schools was 11.4 
and for comparison schools was 6.9. 

Exhibit 3.1 displays the number of ELT and matched comparison schools that achieved each level on 
each of the eight criteria as well as the ELT support criterion. ELT schools, on average, scored higher 
than comparison schools on six of eight criteria. Comparison schools scored, on average, higher than ELT 
schools on the school-wide academic focus and school leadership criteria. Comparison schools scored 
very low (between Level 0 and Level 1) on how core academic time was used, enrichment time, and 
teacher leadership and collaboration time criteria (averages of 0.47, 0.44, and 0.53, respectively). ELT 
schools on average scored lowest (between Level 0 and Level 1) on the school-wide academic focus 
criterion (0.78). Differences were not tested for statistical significance, so the meaning of any differences 
between schools or groups of schools is unclear. 
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Exhibit 3.1: Summary of Implementation Index Levels of Schools, by ELT Status and Index 
Criteria, 2010-11 
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Exhibit 3.1: Summary of Implementation Index Levels of Schools, by ELT Status and Index 
Criteria, 2010-11 


ELT Support 



EXHIBIT READS: On the implementation index, 13 ELT schools and 12 matched comparison schools scored Level 0 on 
Criterion 1: School-wide Academic Locus. 

Sources: Abt Associates’ Interviews ofMA ELT and Matched Comparison School Principals, Spring 2011; Abt Associates’ 
Surveys ofMA ELT School Teachers, Spring 2011; Abt Associates’ Surveys of Matched Comparison School Teachers, Spring 
2011 . 

Sample: 18 ELT schools and 19 matched comparison schools with the exception of Criterions 2 and 3, for which the sample is 17 
ELT and 18 matched comparison schools and Criterions 1, 3 (in addition to time), 4, 5, and 6, for which the sample is 17 ELT 
and 16 matched comparison schools. 


Many comparison schools appeared to be implementing at least some of the key components that are 
considered core expectations of the ELT initiative. Seven comparison schools met Level 3 on the school 
leadership criterion. Lour comparison schools met Level 3 on the school-wide academic focus criterion. 
However, more ELT schools also met Level 3 thresholds across the index criteria. Of note, 9 ELT schools 
met Level 3 on the time component of the enrichment criterion. 

Pie charts depicting which level each ELT and comparison school met for each criterion are shown in 
Exhibits 3.2 and 3.3. A comprehensive matrix that includes the scores for each anonymous school along 
with additional summary data by level of implementation can be found in Appendix C. 
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Exhibit 3.2: ELT School Implementation Pie Charts, by Index Criterion, 2010-11 
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EXHIBIT READS: On the implementation index, School J scored a Level 3 for the Academic Focus criterion. 

Notes: School IDs (letters) have been randomly assigned in order to present school-level implementation data anonymously. Pie charts are organized in descending order of total score 
from left to right. 

The color scheme for this Exhibit is deliberately distinct from other exhibits to enhance readability in both color and black and white. 

Sources: Abt Associates’ Interviews ofMA ELT and Matched Comparison School Principals, Spring 2011; Abt Associates’ Sutyevs ofMA ELT School Teachers, Spring 2011 
Sample: 18 ELT schools with the exception of Criterions 2 and 3, for which the sample is 17 ELT schools. 
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Exhibit 3.3: Matched Comparison School Implementation Pie Charts, by Index Criterion, 2010-11 
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EXHIBIT READS: On the implementation index, School 32 scored a Level 3 for the Academic Focus criterion. 

Notes: School ID numbers have been randomly assigned in order to present school-level implementation data anonymously. Pie charts are organized in descending order of total score 
from left to right. 

The color scheme for this Exhibit is deliberately distinct from other exhibits to enhance readability in both color and black and white. 

Sources: Abt Associates’ Interviews ofMA ELT and Matched Comparison School Principals, Spring 2011 ; Abt Associates’ Surveys of Matched Comparison School Teachers, Spring 
2011 . 

Sample: 19 matched comparison schools with the exception of Criterions 2 and 3, for which the sample is 18 schools and Criterions 1, 3 (in addition to time), 4, 5, and 6, for which the 
sample is 16 schools. 
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Chapter 4. Detailed Implementation Findings 


This chapter describes in depth how the ELT schools implemented core ELT components during the 
2010-11 school year. These components include: expanding the length of the school day, additional time 
and supports for core academics, implementation of a school-wide academic focus, enhanced enrichment 
activities, and opportunities for teacher leadership and collaboration. Implementation is described across 
ELT schools as well as by school configuration when relevant. School configuration refers to schools 
serving different grade levels. For the puiposes of this report, schools are generally grouped as elementary 
or middle schools. Schools serving grades K-4, K-5, K-6, and K-8 are considered elementary schools. 
Schools serving grades 4-7, 5-8, 6-8, and 7-8 are considered middle schools. The chapter does not 
describe cohort-level findings, as the number of schools within cohort varies substantially, as do the grade 
level configurations within cohort, thereby rendering cohort-level differences less meaningful. (See 
Appendix D for more specific information on cohort-specific findings). 

Describing how schools have implemented ELT components allows the study to assess whether the 
initiative overall has been able to translate core principles into operational practice and whether the 
processes outlined in the conceptual model are demonstrably in place. The chapter also includes brief 
descriptions of matched comparison schools’ practices related to the ELT core components based on 
principal interviews in textbox format. Descriptions of teacher and student survey responses about ELT 
implementation are limited to ELT schools. Note that no side-by-side presentations of ELT and matched 
comparison schools’ survey data appear in this chapter; Chapters 5 and 6 present statistical comparisons 
of implementation and outcomes at ELT and matched comparison schools. 

The findings reported in this chapter draw on principal interview as well as teacher and student survey 
data. Unless otherwise stated, reported findings from teachers and students represent responses from 
multiple teachers and students, whereas school-level responses are generally from individual principals. 
Key findings are presented at the beginning of each section. 


Time Use in ELT Schools 

Key Findings 

• The ELT school day was just under eight hours, on average, in 2010-11. 

• On average, almost five of the nearly eight hours of a typical school day were allocated to core 
academics (English Language Arts (ELA), math, science, and social studies). Twenty more 
minutes per day, on average, were allocated to core academics in 5 th than in 8 th grade. 

• Overall, the plurality of time in an ELT school day was allocated to ELA, followed by math, then 
science and social studies. Specifically, of the five hours allocated to core academics: 

- The amount of time scheduled for English Language Arts (ELA) was 1 hour and 45 minutes, 
on average. 

- Nearly 90 minutes were allocated to math instruction, on average. 

- An average of nearly 1 hour was scheduled for science and 45 minutes to social studies per 
day. 
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• Time allocations for core subjects varied somewhat by grade. Specifically, 

- An average of about 45 minutes more each day was allocated to ELA in 5 th than in 8 th grade. 

- Slightly more time (12 minutes, on average) was allocated to math in 5 th than in 8 th grade. 

- About 20 fewer minutes were allocated to science and 15 minutes fewer to social studies in 
5 th than 8 th grade, on average. 

• While there are broad core principles guiding ELT implementation, ELT schools have flexibility 
in how they implement core components. As in past years, schools varied considerably in how 
they allocated time to various instructional activities. 


This section first describes the total amount of time within ELT school days and then discusses time 
allocations to instructional activities and core academic subjects. Next, it describes teachers’ instructional 
practices and the types of supports provided to students. 

Overall Length of the ELT School Day 

As part of their participation in the 
Massachusetts ELT initiative, schools 
were expected to add 300 hours to their 
schedules over the course of the 2010- 
1 1 school year; this increase in time is 
over 1.5 additional hours per school 
day. 35 According to principal responses 
for the target grades, 36 overall, ELT 
schools’ day was just under 8 hours in 
2010-1 1; on average, ELT schools 
started between 7:30 and 8:00 in the 
morning and ended between 3:30 and 
4:00 in the afternoon. Interviews with 
principals at ELT and matched 
comparison schools revealed that ELT 
schools’ days were consistently longer than those at schools without ELT. 

As Exhibit 4. 1 shows, the average ELT school day ranged from about seven-and-a-half to eight-and-a- 
half hours in length. The school day was about 15 minutes longer, on average, for 8 th than 5 th graders, or 
over an hour more each week. In Exhibit 4. 1 , the vertical line represents the full distribution, running 
from the minimum on the bottom, to the maximum, on the top, of the number of hours reported by 
principals for the typical 5 th and 8 th grade students in their schools. The box represents the middle 50 
percent of the distribution, the line through the middle of the box represents the median, and the diamond 
represents the mean value. The lines extending from the bottom and the top of the box represent the 


35 ELT schools were given the option of adding days to their school year but have thus far chosen to add hours 
within the existing school year, which averages 180 days. 

36 Time data include 17 ELT schools (12 5 th and 1 1 8 th grades for a total of 23 data points). Time data analyses 
exclude schools that have neither 5 th nor 8 th grade students (n=l). . 


The table below illustrates what various daily time increments 
represent when aggregated to weekly and annualized levels. 


Difference in Amount of Time Allocated 

Daily 

Weekly 

Yearly 

10 minutes 

50 minutes 

30 hours (one week) 

15 minutes 

75 minutes 

45 hours (one-and-a-half 
weeks) 

20 minutes 

100 minutes 

60 hours (two weeks ) 

30 minutes 

2.5 hours 

90 hours (three weeks ) 

45 minutes 

3.75 hours 

1 35 hours (four weeks) 

60 minutes 

5 hours 

181 hours (six weeks ) 

100 

minutes 

8 hours, 20 
minutes 

300 hours (ten weeks) 
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bottom and top quartiles (from the minimum to the 25 th percentile and from the 75 th percentile to the 
maximum, respectively). 


Exhibit 4.1 : Range in Length of Time Allocated to School Day in ELT Schools, by Grade, 
2010-11 


09:00 i 
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EXHIBIT READS: The figure presents the range of the length of the school day across all ELT schools, as well as for a 
typical 5 th and 8 th grade student. The shaded box represents the middle 50 percent — from the 25 th to the 75 th percentiles — of 
the distribution of school day length, and the lines to the bottom and top represent the bottom and top quartiles, respectively, 
extending to the minimum and maximum values. The line in the middle of the shaded box represents the median value for 
each distribution, and the diamond shape represents the mean value. The median value for all ELT students is about 7 hours, 
46 minutes; for 5 th grade is about 7 hours, 39 minutes; and for 8 th grade is 7 hours, 46 minutes. 

Source: Abt Associates’ Interviews of MA ELT School Principals, Spring 2011, Item 9a through 9e. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 l-8 th grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5 th or 8 th grades. 


In 2010-1 1, students’ opinions of the length of the school day were more negative than positive, based on 
survey data. Exhibit 4.2 illustrates that, overall, about half of students reported disliking the length of the 
day. Student opinions about the length of the school day were consistent across grade configuration (not 
shown). 
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Exhibit 4.2: Student Satisfaction with Length of School Day, 2010-11 



□ Like/Strongly Like ■ Neutral □ Strongly Dislike/Dislike 


EXHIBIT READS: In spring 2011, across all ELT schools, 22 percent of students reported that they like or strongly like the 
length of the school day. 

Source: Abt Associates’ Surveys of MA ELT School Students, Spring 2011, Item 5. 

Sample: 2,300 students from ELT schools. Nonresponse rate was 1.4 percent. 


ELT School Time Allocated to Different Instructional Activities 

Given the longer school day, ELT schools had different options about how to allocate additional time 
across the following categories of instruction and other activities: 

• Core academic subjects: ELA, math, science, and social studies; 

• Dedicated academic support, an extra block of academic time that targeted specific academic 
skills based on student needs; 

• Specials , which included physical education, art, music, computers, and foreign language ; 37 

• Enrichment classes, which were academic or non-academic in nature, with the goal of building 
student interests and skills and deepening engagement with learning as well as their relationships 
with the individuals (teachers or staff from community partner organizations) they learned with; 
and 

• Other activities, which included lunch, recess, snack, homeroom, and transitions. 


37 Four of 17 ELT schools included in the analysis of time data had required academic courses other than the four 
core subject areas. Generally, such courses included foreign language (i.e., Spanish, Portuguese, Chinese, and 
French) and technology courses. . Since all of the students within grade level attended these courses, and since 
so few schools offered these courses, this category was combined with the specials category. Note that these 
four ELT schools also offered traditional specials: physical education, art, music, and computers. 
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Panel 1 of Exhibit 4.3 presents the amount of time and Panel 2 presents the percentage of time ELT 
schools allocated to each of the above activities per day. 38 As noted above, over half of the nearly eight 
available hours each day (about 61 percent of available time) was allocated to core academics. About an 
hour each day was allocated in ELT schools to specials 39 and about an hour to the “other” category, which 
included all administrative and lunch/recess activities (5 percent and 10 percent of the day, respectively). 
About a half hour each day was allocated to enrichment and a half hour to academic support (5 percent 
and 6 percent of the day, respectively). 

The amount of time allocated to core academics was about 20 minutes more each day for 5 th graders than 
for 8 th graders, on average, corresponding to 4 hours and 55 minutes for 5 th graders and 4 hours and 37 
minutes for 8 th graders per week. Of note, the amount of time allocated to 5 th graders for enrichment and 
academic support was less, on average, than for 8 th graders (about 20 minutes each for 5 th graders, and 30 
minutes each for 8 th graders). Across both grades, about the same amount of time was allocated for 
specials and for “other” activities (about 60 minutes each). 


38 According to principals, not all types of instmction were necessarily offered daily; however, for purposes of 
summarizing how schools allocated time, amounts of time for such activities were converted into daily metrics. 

39 As mentioned above, four ELT schools had what could be termed “non-core academic courses”; at those 
schools, an average of 40 minutes per day was allocated to “non-core academics” as well as an average of one 
hour per day to traditional specials. 
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Exhibit 4.3: Amount and Percentage of Time Allocated to Instructional Activities in ELT 
Schools, Per Day, by Grade, 2010-11 

Panel 1 



□ Core DOther DSpecials ■Academic Support DEnrichment 


Panel 2 

ELT Overall 


ELT 5th 


ELT 8th 



EXHIBIT READS (Panel 1): In spring 2011, across all ELT schools, 4 hours and 46 minutes per day were allocated to core 
academics. 

EXHIBIT READS (Panel 2): In spring 2011, across all ELT schools, 61 percent of the school day was allocated to core 
academics. 

Source: Abt Associates’ Interviews ofMA ELT School Principals , Spring 2011, Item 11. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 1 -8 th grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5 th or 8 th grades. 
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The data described above represent aggregated findings across all ELT schools; the displays below 
illustrate variation across individual schools. ELT schools received school-specific guidance and 
flexibility about how to implement specific ELT components, and therefore it is not surprising that 
schools varied considerably in how they allocated time to various instructional activities. Exhibit 4.4 
summarizes the range of time allocated to each broad category of instruction for 5 th and 8 th graders in 
2010-11. Exhibit 4.5 presents data on individual school-level allocations across these categories for 5 th 
and 8 th graders. The amount of time allocated to enrichment and academic support ranged from 0 to 41 
minutes and 0 to 68 minutes per day, on average, for enrichment, and from 0 to 57 and from 0 to 64 
minutes per day for academic support (for 5 th and 8 th graders, respectively). 40 In general, there was less 
variation across schools in how time was allocated in 5 th grade than in 8 lh grade. Of note, in 2009-10, two 
5 th and two 8 th grades reported allocating no time for enrichment, and five 5 th and two 8 th grades reported 
no academic support. 

For the purposes of presenting information about ELT implementation in this chapter, each school is 
identified by a randomly assigned (unique) letter, and the individual school displays use these identifiers. 
School-by-school results presented in subsequent chapters do not use the same identification scheme as 
used here. 


40 According to principals, one ELT school (serving both 5 lh and 8 th graders) did not provide any enrichment to its 
students; another ELT school (also serving both 5 th and 8 th graders) provided enrichment to 5 th but not to 8 th 
graders. Also, two ELT schools (serving both 5 th and 8 lh graders) and two ELT schools (serving only 5 th 
graders) did not have a discrete block during which students were provided academic support. 
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Exhibit 4.4: Range in Amount of Time Allocated to Instructional Activities in ELT Schools Per 
Day, by Grade, 2010-11 
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EXHIBIT READS: The figure presents the range of the amount of time, per day, for several types of instruction: core academic 
subjects, academic support, special, enrichment, and all other types of instruction for a typical 5 lh and 8 lh grade student. The 
shaded box represents the middle 50 percent — from the 25 th to the 75 th percentiles — of the distribution of school day length, and 
the lines to the bottom and top represent the bottom and top quartiles, respectively, extending to the minimum and maximum 
values. The line in the middle of the shaded box represents the median value for each distribution, and the diamond shape 
represents the mean value. The median value for core instruction in 5 lh grade is about 5 hours, 3 minutes and the median value for 
8 th grade is about 4 hours, 35 minutes. 

Source: Abt Associates’ Interviews of MA ELT School Principals, Spring 2011, Item 11. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 l-8 lh grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5 th or 8 th grades. 
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Exhibit 4.5: Amount of Time Allocated to Instructional Activities in ELT Schools Per Day, by 
School and Grade, 2010-11 



□ Core aOther DSpecials ■Academic Support □Enrichment 



8th 


EXHIBIT READS: In spring 2011, the largest amount of time in School I for 5 th grade was allocated to core academics (the 
blue color block), which corresponds to 4 hours and 10 minutes of the 7 hour, 40 minutes school day. 

Notes: School IDs have been randomly assigned in order to present school-level implementation data anonymously. 

Schools are presented in ascending order by amount of time allocated to core academics. 

Source: Abt Associates’ Interviews ofMA ELT School Principals, Spring 2011, Item 11. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 l-8 lh grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5 th or 8 lh grades. 
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ELT School Time Allocated to Core Academics 

This section examines time allocation for core academics in ELT schools, drawing from principal- 
provided information on school schedules for prototypical 5 th and 8 th grade students. A core expectation 
for ELT implementation is that each ELT school “[use] additional time in order to accelerate learning in 
core academic subjects.” 41 Key indicators for this ELT component include: 1) the school schedule for 
student learning provides adequate time for the delivery of instruction, and 2) the schedule provides 
adequate time for individualized support, be it remedial skill development or skill acceleration. 

On average, ELT schools allocated nearly five of the eight available hours each day (about 60 percent of 
available time) on core academics. The majority of core instructional time was scheduled for ELA 
instruction, followed by math, and the remainder of the time was split about evenly between science and 
social studies, with slightly more time allocated to science. 

Panel 1 of Exhibit 4.6 presents the amount of time, and Panel 2 presents the percentage of time ELT 
schools allocated to four core academic subjects per day. 42 Overall, an average of one hour and forty-five 
minutes per day were devoted to ELA, just under 90 minutes to math, nearly an hour to science and about 
three-quarters of an hour to social studies. 

Generally, a larger proportion of core instructional time was allocated to ELA for 5 th than 8 th grade 
students, corresponding to 42 and 30 percent of instructional time, respectively. Just under one-third of 
instructional time in each grade level was devoted to math. On average, less core instructional time for 5 th 
than for 8 th grade was devoted to science and social studies (about 1 5 percent and 20 percent in each 
subject, respectively). 

There were grade level differences in how much time was allocated to core academic subjects. Overall, 
the total amount of time allocated to core academics was about 20 minutes more for 5 th than for 8 th grade. 
On average, nearly 45 minutes more per day were allocated to ELA in 5 th than in 8 th grade (2 hours and 5 
minutes and 1 hour and 23 minutes, respectively). On average, slightly more time, about 12 minutes per 
day, was allocated for 5 th than for 8 th grade to math (1 hour and 28 minutes and 1 hour and 16 minutes, 
respectively). About 20 minutes fewer were allocated to science and 15 minutes fewer to social studies in 
5 th than 8 th grade. 


41 ELT Expectations for Implementation, Mass 2020 website: http://www.mass2020.org/node/14. Retrieved 
12/15/11. The Expectations were developed by ESE, Mass 2020, and Focus on Results 

42 According to principals, not all types of instruction were necessarily offered daily; for purposes of summarizing 
how schools allocated time, amounts of time for such activities were converted into daily metrics. 
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Exhibit 4.6: Amount and Percentage of Time Allocated to Core Academics in ELT Schools Per 
Day, by Grade, 2010-11 

Panel 1 



□ ELA ■Math □ Science □ Social Studies 


Panel 2 

ELT Overall 


ELT 5th 


ELT 8th 



EXHIBIT READS (Panel 1): In spring 2011, across all ELT schools, 1 hour and 45 minutes per day were allocated to 
instruction in ELA. 

EXHIBIT READS (Panel 2): In spring 2011, across all ELT schools, 37 percent of the time allocated to core academics was 
allocated to instruction in ELA. 

Source: Abt Associates' Interviews of MA ELT School Principals, Spring 2011, Item 11. 

Sample: Time data analyses include 17 ELT schools (12-5* and 11-8* grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5* or 8* grades. 
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Exhibit 4.7 summarizes the range of time allocated to each core academic subject in 5 th and 8 th grade, and 
Exhibit 4.8 presents data on individual school-level allocations of time across core academic subjects for 
these grades. While there was some variation by school, 5 th and 8 th grade time allocations were generally 
similar to the overall cross-school allocation. 


Exhibit 4.7 : Range in Amount of Time Allocated to Core Academics in ELT Schools Per Day, by 



ELA 5th ELA 8th Math 5th Math 8th Science Science Social Social 

5th 8th Studies 5th Studies 8th 


EXHIBIT READS: The figure presents the range of the amount of time, per day, for instruction in core academic subject 
areas, including English Language Arts (ELA), Math, Science, and Social Studies for a typical 5 th and 8 th grade student. The 
shaded box represents the middle 50 percent — from the 25 th to the 75 th percentiles — of the distribution of school day length, and 
the lines to the top and bottom represent the top and bottom quartiles, respectively, extending to the minimum and maximum 
values. The line in the middle of the shaded box represents the median value for each distribution, and the diamond shape 
represents the mean value. The median value for ELA instruction in 5 th grade is about 2 hours, 7 minutes and the median value 
for 8 lh grade is about 1 hour, 26 minutes. 

Source: Abt Associates’ Interx’iews ofMA ELT School Principals, Spring 2011, Item 11. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 l-8 lh grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5 th or 8 th grades. 
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Exhibit 4.8: Amount of Time Allocated to Core Academics in ELT Schools Per Day, by School 
and Grade, 2010-11 
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EXHIBIT READS: In spring 2011, the largest amount of time in School I for 5 th grade was allocated to instruction in ELA (the 
blue color block), which corresponds to about 1 and a half hours of the 4 hours, 10 minutes allocated to core academics. 

Notes: School IDs have been randomly assigned in order to present school-level implementation data anonymously. 

Schools are presented in ascending order by amount of time allocated to ELA. 

Source: Abt Associates' Interviews of MA ELT School Principals, Spring 2011, Item 11. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 l-8 lh grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5 th or 8 lh grades. 
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While there were some differences in the proportion of teachers who reported satisfaction with the 
amount of instructional time available for each core academic subject, overall, satisfaction with 
instructional time in core subject areas was high (70 percent or more teachers reporting satisfaction). 
Teacher satisfaction with the amount of instructional time available for core academics was higher for 
ELA and math than for science and social studies, according to survey data. As shown in Exhibit 4.9, 
overall, 90 percent or more of teachers were satisfied with the amount of time available for instruction in 
ELA and math. About 80 percent of teachers were satisfied with time available for science and three- 
quarters (7 1 percent) with time available for social studies. A higher proportion of middle school than 
elementary teachers reported satisfaction with the amount of instructional time available for instruction 
for science and for social studies (86 and 76 percent, and 75 and 68, respectively). 

As shown in Exhibit 4.9, however, overall, less than half of teachers (42 percent) were satisfied with the 
amount of time available during the day for physical activity. Elementary school teachers were more 
satisfied than middle school teachers with the amount of time during the day for physical activity (52 and 
28 percent, respectively). 43 


Exhibit 4.9: Teacher Satisfaction with Instructional Time Available for Core Academics and 


Physical Activity, by School Configuration, 2010-11 



Percent of Teachers 

Overall 

Elementary 

Middle 

Time available for instruction in math. 

93% 

94% 

90% 

Time available for instruction in ELA. 

90 

92 

87 

Time available for instruction in science. 

81 

76 

86 

Time available for instruction in social studies. 

71 

68 

75 

Time available for physical activity. 

42 

52 

28 


EXHIBIT READS: In spring 2011, across all ELT schools, 93 percent of teachers reported that they are satisfied or 
very satisfied with the amount of time available for instruction in math. 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 10b, 10a, 10c, lOd and lOg. 

Sample: 918 teachers from ELT schools. Nonresponse rates across items ranged from 8.2 to 24.1 percent. 


Student reports indicate some subject-specific variation in the amount of time students need to learn a 
given subject (see Exhibit 4.10). For example, about two-thirds of students reported that they need more 
time in Specials and other non-academic activities, about half of student respondents indicated that they 


43 All time data have been presented by grade (5 th and 8 th ), whereas survey responses are calculated by school 
configuration (elementary and middle). The time data analyzed by grade revealed no differences between 5 lh 
and 8 th grades in the average amount of time allocated to specials and “other” activities (e.g., homeroom, 
recess, lunch, and transitions). Time data calculated by school configuration, however, suggest that 15 more 
minutes per day are allocated to specials in elementary than in middle schools (60 vs. 45 minutes per day, 
respectively). Likewise, time data calculated by school configuration suggest that 10 more minutes per day are 
allocated to “other” activities in elementary than middle schools (52 vs. 42 minutes per day, respectively). 
Differences in elementary and middle school teachers’ satisfaction with the amount of time during the day for 
physical activity might reflect the differences in time allocated to specials and other activities. That said, even 
though elementary teachers were more satisfied with time available during the day for physical activity than 
middle school teachers, overall satisfaction is low. 
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need more time to learn science, math, and social studies well, and fewer students reported that they need 
more time to learn reading, vocabulary, and writing. 


Exhibit 4.10: Student Perceptions of Whether They Need More Time for Core Academics and 
Specials, 2010-11 


I need more time in Specials and other non-academic 
activities. 

I need more time to learn Math well. 
I need more time to learn Science well. 
I need more time to learn Social Studies well. 
I need more time to learn Writing well. 
I need more time to learn Vocabulary well. 
I need more time to learn Reading well. 


20 


] 51 
I 50 


47 


] 41 


] 41 


] 34 


65 


40 60 

% True 


80 


100 


EXHIBIT READS: In spring 2011, across ELT schools, 65 percent of students reported that they needed more time in Specials 
and other non-academic activities. 

Source: Abt Associates’ Surveys of MA ELT School Students, Spring 2011, Item 10a through lOg. 

Sample: 2,300 students from ELT schools. Nonresponse rates across items ranged from 1.6 to 2.0 percent. 


Effects of Longer Day on Instructional Practice of ELT Teachers 

Adequate time for instruction is likely to be constructive for student learning only if the time is used 
effectively. Administrators and teachers in ELT schools are expected to identify and implement effective 
instructional practices school-wide. Implementation typically requires substantial support to teachers from 
principals, coaches, and school-based professional development that includes feedback through frequent 
classroom visitation, modeling of effective practices, and assistance in analyzing and using performance 
data. Teachers can also improve practices by sharing with and learning from their peers. Teacher reports 
on the amount and quality of instructional time spent with students are described in this section; a 
subsequent section presents findings on supports provided to ELT teachers for instructional improvement. 

Survey responses indicate that teachers in ELT schools overwhelmingly agreed (9 1 percent) that they 
spent sufficient instructional time with their students for them to be able to learn; teachers also generally 
agreed that students had sufficient time to learn beyond the minimal requirements (76 percent) Teachers 
also consistently reported that they were able to use a variety of instructional strategies (e.g., project- 
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based, small-group learning) 44 , differentiate instruction for students of different abilities, cover the 
amount of instructional material their school expected, and cover the amount of instructional material 
their students needed to learn (92, 89, 82 and 74 percent, respectively). Further, the large majority of 
teachers (80 percent) reported that the length of the school day allowed them to accomplish their teaching 
goals. More elementary than middle school teachers reported that they were able to cover both the amount 
of instructional material their school expected and that their students needed to learn (87 and 75 percent 
and 81 and 65 percent, respectively) (not shown). 


Academic Support for Students in ELT Schools 

Another core ELT expectation is that adequate time be allocated for the provision of individualized 
support and that this time be tailored to individual student needs. Academic support is intended to build 
on the quality instruction and foundational skills established in core courses, and is not meant to replace 
such courses. This section focuses on dedicated blocks of time to academic support time distinct from 
academic support provided in a more integrated manner during other classes (i.e., “push in” support). 

Like academic support classes, homework can also allow students to practice what they learn in their 
academic classes and further their skills. Information on homework policies is presented in Appendix D. 

According to survey data, less than half of students (43 percent) reported needing extra help to learn 
reading, math, science, and/or social studies. Just over two-thirds (67 percent), however, reported that 
they needed more time to learn subjects that were hard for them. These data suggest that student 
perceptions align with core expectations regarding academic support; tailored, individualized academic 
support is essential if students are to progress in subject areas with which they struggle. 

Time Allocated to Academic Support 

Principals at more than three-quarters of the ELT schools (14 of 18) 45 reported that they provided 
dedicated blocks of academic time targeting specific academic skills. All eight middle schools, whereas 
just over half of the elementary schools provided dedicated academic support (6 of 10). At the majority of 
the schools that provided dedicated academic support (12 of 14), academic support time was provided to 
all students (at the other two it was provided to half or nearly two-thirds of the students). Across the 14 
schools that provided dedicated academic support, on average three hours (of the 40 hour week) were 
allocated to dedicated academic support (not shown). More time (about 45 minutes) was allocated to 
academic support each week for 8 th than 5 th grade (three hours and 25 minutes and two hours and 39 
minutes, respectively) (not shown). 

The data described above represent aggregated findings across all ELT schools; the displays below (see 
Exhibit 4.1 1) indicate considerable school-level variation in time allocated to academic support for 
students both within and across grade level. Interestingly, the average amount of time allocated to 
dedicated academic support increased by an hour for 5 th grade, and nearly an hour for 8 th grade since 
2009-10 (from 1 hour and 39 minutes to 2 hours and 39 minutes, and from 2 hours and 38 minutes to 3 
hours and 25 minutes, respectively). 

44 According to survey data, slightly less than two-thirds of students (62 percent) reported that they needed more 
time to do project-based and hands on activities in classes; these data were consistent across school 
configurations. 

45 At one school dedicated academic support was provided to the 8 lh but not to the 5 lh graders attending the school. 
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Exhibit 4.1 1 : Amount of Time Allocated to Academic Support in ELT Schools Per Week, by 
School and Grade, 2010-11 


ELT 5th 



ELT 8th 



EXHIBIT READS: In spring 2011, School N allocated 0 hours per week for academic support time for 5 lh graders. 

Notes: * Averages are calculated only among schools providing academic support. 

School IDs have been randomly assigned in order to present school-level implementation data anonymously. 

Schools are presented in ascending order by amount of time allocated to academic support. 

Source: Abt Associates' Interviews ofMA ELT School Principals, Spring 2011, Item 11. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 l-8 lh grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5 th or 8 th grades. 


According to survey responses, about three-quarters of teachers agreed that all students had access to 
dedicated academic support and that the length of time for and frequency of dedicated academic support 
allowed enough time for meaningful instruction (72 and 78 percent, respectively). 
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Content, Student Grouping, and Grading of Academic Support Classes 

According to principals, during the 2010-11 school year, academic support classes offered both 
remediation and acceleration to students in math and ELA (see Exhibit 4.12). Few academic support 
offerings focused on science, and almost none focused on social studies. The focal subjects in academic 
support classes were similar across school configuration (not shown). 


Exhibit 4.12: Focal Subjects in Academic Support in ELT Schools, 2010-11 


ELT Schools 

Providing Dedicated Academic Support (N=14) 



Math 

ELA 

Science 

Social Studies 

Remediation 

14 

12 

2 

2 

Acceleration 

8 

6 

4 

1 


EXHIBIT READS: In spring 2011, across the 14 ELT schools in which principals reported having a dedicated block of 
academic support, 14 reported that remediation in math was a focal subject of this academic support. 

Source: Abt Associates’ Interviews ofMA ELT School Principals, Spring 2011, Item 12i. 


One aspect of high quality academic support time is the deliberate grouping of students and that grouping 
decisions take into account student performance data as well as teacher input. The expectation for 
academic support is that effective grouping is organized by ability level or by specific target skill, and not 
by homeroom, grade level, or happenstance. 46 Additionally, these groupings should be revisited over the 
course of the year with the flexibility to reassign students according to their changing needs. 

In 12 of 14 schools that provided academic support, principals explained that students were grouped by 
ability level in the specific subject in which they needed support; the other two, both elementary schools, 
reported that students were kept with their regular homerooms during academic support time. Student 
assignment to academic support was based on academic performance (13 of 14) and secondarily, on 
teacher recommendations (nine of 14); other considerations included grade level, Limited English 
Proficiency (LEP) status, and parent concerns. In 10 of 14 schools, students were reassigned to academic 
support classes throughout the course of the year. Principals typically reported that reassignment 
happened at least quarterly; reassignment was linked to timing of assessments used to determine 
reassignment. Principals reported that the amount of academic support students received was based on the 
schedule (e.g., four 45-minute blocks per week). 

According to survey responses, the majority of teachers agreed that students were appropriately grouped 
in academic support classes and that their performance in academic support was assessed regularly (78 
and 81 percent). 


46 Using Increased Learning Time for Individualized Instruction and Tiered Support, Massachusetts 2020 

presentation on October 15, 2010 and Framework for Assessing Tiered Interventions and Academic Support, 
Massachusetts 2020, October 15, 2010. 
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Another core tenet is that academic support be led by qualified, professional staff who are knowledgeable 
about the core curriculum and about instructional strategies that support struggling students. 47 Decisions 
about which teachers should teach which content and to which students should be made deliberately and 
with consideration for performance data as well as for teacher and student personalities. Partner staff may 
be involved in the provision of academic support but are expected to have experience or be trained in this 
area. 

In 2010-1 1, principals reported that teachers were generally assigned to dedicated academic support 
classes within their own subject area and grade level. A few middle school principals explained that 
teachers volunteered to teach academic support classes. As shown in Exhibit 4.13, principals reported that 
academic support classes were taught by regular classroom teachers in all 14 of the schools in which 
academic support was provided. Principals also reported that special education teachers were responsible 
for teaching academic support in nine of 14 schools. In five schools, principals indicated that, academic 
support classes were taught by math or ELA coaches. In fewer schools, paraprofessionals and other staff 
(e.g., tutors, masters candidates) were hired/placed specifically to provide academic support. Overall, all 
principals reported that all staff providing academic support were licensed in the respective content areas; 
when academic support was provided by non-specialists (in that content area), principals described 
efforts to support collaboration between academic support providers and colleagues with content 
expertise who were also teaching academic support periods. 


Exhibit 4.13: Background of Academic Support Instructors in ELT Schools, 2010-11 



ELT Schools Providing Dedicated Academic 
Support 


Overall 

(N=14) 

Elementary 

Schools 

(N=6) 

Middle 

Schools 

(N=8) 

Regular classroom teachers 

14 

6 

8 

Special education teachers 

9 

4 

5 

Math or ELA coaches 

5 

4 

1 

Staff specifically hired to teach dedicated academic 
support classes 

3 

1 

2 

Paraprofessionals 

3 

2 

1 


EXHIBIT READS: In spring 2011, across the 14 ELT schools in which principals reported having a dedicated block of 
academic support, all 14 reported that regular classroom teachers teach these academic support classes. 

Note: Responses add up to more than 14 because respondents were allowed to select multiple responses and because some of 
the categories overlap (e.g., regular classroom teachers might also have subject matter expertise). 

Source: Abt Associates’ Interviews ofMA ELT School Principals, Spring 2011, Item 12f. 


Overall, according to survey responses, the majority of teachers agreed that dedicated academic support 
classes were taught by individuals knowledgeable both about the core curriculum and about instructional 


47 Using Increased Learning Time for Individualized Instruction and Tiered Support, Massachusetts 2020 

presentation on October 15, 2010 and Framework for Assessing Tiered Interventions and Academic Support, 
Massachusetts 2020, October 15, 2010. 
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strategies to support struggling students (88 and 84 percent). Teacher responses were consistent across 
school configuration. 

As with core academic classes, it is expected that teachers will be provided the support to improve their 
instruction in academic support classes through regular 
classroom visitation by administrators and teacher 
leaders to provide feedback. Less than half of teachers 
(46 percent), however, agreed that administrators 
regularly visited academic support classes to provide 
feedback. 

Another important component of quality academic 
support is that teachers have opportunities to 
collaborate, both to discuss student progress as well as 
to share instructional techniques. Overall, about one- 
third of teachers (36 percent) reported coordinating 
their instruction with academic support personnel 
during common planning time (not shown). Of those, 
about a quarter indicated that engaging in this activity 
helped their teaching. These patterns did not differ by 
school configuration. 

Use of Data in ELT Schools 

In addition to state-mandated student achievement 
measures, ELT schools used a variety of other 
assessment tools during the 2010-11 school year. The 
most commonly used assessment tool was the DIB ELS 
(Dynamic Indicators of Basic Early Literacy Skills) 
which was used in 10 schools. In terms of content 
focus, science was assessed more often by these 
additional tools at the middle school level (6 of 8 
schools) than at the elementary school level (4 of 10 
schools). Similarly, social studies was assessed at most 
middle schools (6 of 8 schools) and 2 of 10 elementary 
schools. 


Additional assessment data were reviewed and used by 
ELT school staff with varying frequency. Of the 18 ELT schools, twelve schools used assessment data 
weekly, three schools monthly, and three schools quarterly. Assessment data were used by nearly all 
schools to strengthen instruction (17 schools) and to group students for academic support (16 schools). 


Academic Support for Students in 
Matched Comparison Schools 

Principals at six matched comparison schools 
indicated that they provided dedicated 
academic support time to all or most of their 
students. Several others indicated that they 
provided a subset of students with extra 
support, or that academic support was 
integrated into core academic classes. Twelve 
principals indicated that they used test scores 
to identify students who were in need of extra 
support, and six indicated that teacher 
recommendations identified students who need 
academic support. 

At nine comparison schools, principals 
indicated that academic support was taught by 
regular classroom teachers. Four principals 
specified that teachers provide academic 
support in their respective area of content 
expertise, two of whom further specified that 
the most skilled teachers support the most 
struggling students. At four other schools, 
principals indicated that academic support was 
generally provided by staff who were hired 
specifically to do so, including part-time staff. 

Twelve comparison school principals reported 
that academic support focused on ELA 
remediation, and 13 on math remediation. 

Several comparison schools offered summer or 
vacation programming to students, typically 
those who had failed to pass core academic 
courses during the semester or school year. 
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School-Wide Academic Focus 

Key Findings 

• In the 2010-11 school year, most ELT schools had a school-wide academic focus, according to 
both principals and teachers. Teacher and principal reports of the focus were consistent at 1 1 of 
16 schools. 

• Both teachers and principals reported that the most common focus area was literacy, although 
writing, math, and higher order thinking skills were also common foci. 

• Elementary school teachers reported a literacy-related focus more frequently than middle schools, 
and middle school teachers reported that higher order thinking skills was the focus more 
frequently than elementary schools. 

• Students were most likely to report that the focus was math; however, student reports were rarely 
consistent with those of principals and teachers or internally consistent within a school. 

• At most schools, the focus was posted publicly, often in hallways, teachers’ classrooms, and the 
administrative offices. 

• According to principals, most ELT schools had implemented school-wide instructional practices, 
and the vast majority of teachers reported that their instructional practice was influenced by the 
focus. 

• A substantial majority of teachers reported that they used data specific to the focus area to 
monitor student progress and adjust instructional practices. Most also reported that dedicated 
academic support was influenced by the school-wide focus. 

During the 2010-11 school year, ELT schools were, as in prior years, expected to establish and support a 
school-wide academic focus that “drives instructional improvement and continuous measurable growth in 
student learning throughout the redesigned day and year.” The academic focus area represents an 
instructional need identified on the basis of student assessment data, and that school leadership and staff 
have jointly determined to improve instruction and raise student achievement. The focus should be clear 
and easy to communicate to staff, students, parents, and the community. Schools are expected to align 
ELT implementation with the overall academic focus. 48 Student performance data should be posted in the 
school to track progress toward meeting the focus goals throughout the year, and teachers should utilize 
assessment data to adjust instructional practices accordingly. Schools were asked to consider the 
academic focus as “one thing [they] do excellently, expertly, better than anyone else, in every classroom 
for every student,” and with it, should come built-in accountability. 49 Mass 2020 and Focus on Results 
continued to help schools develop and support their selected focus area(s) through ongoing coaching 
visits and leadership training sessions. 


48 

ELT Expectations for Implementation, Mass 2020 website: http://www.mass2020.org/node/14 Retrieved 
1 1/14/11. The Expectations were developed by ESE, Mass 2020, and Focus on Results. 

49 Instructional Focus: What it is — what it isn ’t, Focus on Results website. 
http://www.focusonresults.net/curriculum/index.html Retrieved 1 1/14/1 1 . 
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Sixteen of 18 ELT school principals reported that their schools had established a school- wide academic 
focus. 50 All middle schools had a focus. The majority of teachers (8 1 percent) reported that their school 
had an academic focus, with no substantial differences by school configuration. Fourteen percent of 
teachers reported not knowing whether their school had an academic focus area. 

Nearly half of the principals discussed their school’s academic focus area(s) as comprising multiple topic 
areas. 51 Seven of 16 principals reported that literacy was a focus, and six reported that writing was a 
focus. Seven reported other foci such as higher order thinking skills, open response questions, or the use 
of instructional tools like T-charts. 52 Three principals reported math, and one reported vocabulary. No 
principals reported that science or social studies was the school’s focus. Some schools used both “student 
friendly” and adult terminology to describe their focus. 

Overall, teacher survey reports about the school’s academic focus were similar to principal reports, and 
are summarized in Exhibit 4. 14. However, teacher reports were not always consistent with principal 
reports within schools. 53 Responses were consistent at 1 1 of 16 schools. The most common focus reported 
by teachers was literacy (36 percent), followed by math (14 percent) and higher order thinking skills (1 1 
percent). Twenty-four percent of teachers reported that the focus was something other than the topics 
listed on the survey. 


50 Given the relatively small number of ELT schools overall, this discussion presents information in terms of 
number of schools rather than percentages. 

51 In interviews, principals responded to an open-ended question regarding their academic focus. Teachers 
reported the academic focus through surveys and were asked to choose one of eight specific topics listed or to 
write in an additional topic if their school’s focus was not listed. 

52 Foci were not always mutually exclusive. For example, a focus on open response questions may overlap with a 
focus on writing/literacy. 

53 To determine within-school consistency, teacher reports from surveys were compared to principal reports from 
interviews for each school. Responses were considered “consistent” if more than 50% of teachers reported a 
topic that aligned with the principal. For this analysis, literacy, writing, and vocabulary were broadly considered 
to be English Language Arts. Since foci were not always mutually exclusive, other foci such as open response 
or T-charts were examined on a case-by-case basis to determine if they should be considered a match. 
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As reported by teachers, school academic focus areas varied somewhat by school configuration (see 
Exhibit 4.14). Elementary school teachers reported a literacy focus more frequently than middle school 
teachers (47 and 23 percent, respectively). Middle school teachers reported higher order thinking skills 
more often than elementary teachers (19 compared to 5 percent). Student survey responses indicate less 
consistent recognition of the academic focus areas identified by principals and teachers (see Exhibit D.3 
in Appendix D) 


Exhibit 4.14: Teacher Reports of Academic Focus, by School Configuration, 2010-11 



Percent of Teachers Reporting 

Overall 

Elementary 

Middle 

Literacy 

36% 

47% 

23% 

Math 

14 

12 

16 

Higher Order Thinking Skills 

11 

5 

19 

Writing 

9 

8 

9 

Vocabulary 

3 

0 

6 

Test Taking Skills 

2 

2 

3 

Science 

0 

1 

0 

Social Studies 

0 

0 

0 

Other 

24 

24 

23 


EXHIBIT READS: In spring 2011, across all ELT schools, 36 percent of teachers reported that literacy was the academic 
focus at their school. 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 19. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 1 percent. 
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According to principals, the academic focus was posted 
publicly at most schools (13 of 16), usually in the 
hallways (12 schools), teachers’ classrooms (11 
schools), and/or administrator’s office (10 schools). 

Less than half of the schools posted the focus on-line or 
in the teachers’ lounge (7 and 6 schools, respectively). 

Other ways that schools spread awareness of the focus 
were through the distribution of parent newsletters, 
posting the focus in the school gym, or reciting the 
focus during school announcements. Three-quarters of 
teachers reported that the focus was posted publicly. 

Principal interviews and site visits revealed that the 
academic focus had moderately more visibility at 
middle schools than elementary schools. For instance, 
the focus was posted in hallways, classrooms, and 
administrative offices somewhat more often at middle 
schools than elementary schools. 

Exhibit 4.15 summarizes teachers’ perceptions about 
the school’s academic focus. Overall, teachers reported 
that the academic focus had a significant influence on 
the school. The vast majority of teachers reported that 
their instructional practice was influenced by the 
school-wide academic focus (95 percent), which was a 
substantial increase from 2009-10 (65 percent). A large 
majority of teachers also reported that they used data 
specific to this year’ s school- wide academic focus area 
to monitor student progress, adjust instructional 
practices, and inform dedicated academic support (each 
at 90 percent). Roughly three-quarters of teachers 
reported that enrichment activities were influenced by 
the academic focus (73 percent), more than the 62 
percent reported in 2009-10. Reports from teachers 
across school configuration were similar. 

Nearly all principals (15) reported that they had school- 
wide instructional practices in place. Examples of such 
practices included the use of tiered instruction, T-charts 
or other instructional tools, specific note-taking 
techniques, reciprocal teaching, and the workshop model in which students present at the end of every 
class. Three ELT principals mentioned that the practices were supported by providing teachers with 
related professional development or training. 


Academic Focus at Matched Comparison 
Schools 

Fifteen of 19 comparison schools reported the 
establishment of a school-wide academic focus in 
2010-11. Comparison school principals, like ELT 
principals, sometimes reported more than one 
focus area, including literacy (11 schools), followed 
by writing (four schools), vocabulary and math 
(three schools each). Two schools had other 
specific foci such as T-charts, open response, and 
student-to-student discourse. Teacher reports of 
the focus were consistent with principal reports at 
12 schools (80 percent of the schools with a focus); 
the proportion of schools with consistency was 
higher than for ELT schools. In no comparison 
schools were student reports about the content of 
the focus consistent with teacher and principal 
reports. 

Most comparison schools with an academic focus 
publicly posted it in at least one location (12 of 15 
schools). More than half of schools posted the 
focus in classrooms (eight schools), and fewer in 
hallways, on-line, in the teachers’ lounge, and in the 
administrative office (five, five, four and four 
schools, respectively). Other locations for the focus 
included conference rooms, communications sent 
home to parents, and on cards taped to each 
student’s desk. 

Fourteen comparison principals reported that their 
schools implemented school-wide instructional 
practices, including practices similar to those used 
at ELT schools, such as specific note-taking 
techniques, word walls, reciprocal teaching, the 
workshop model of instruction, and strategies from 
the Bay State Reading Institute. Five principals 
mentioned providing professional development to 
teachers to support school-wide instructional 
practices. For example, one school trained teachers 
in a reading strategy called QAR (Question Answer 
Relationship) and posted the strategy in all 
classrooms throughout the school. 
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Exhibit 4.15: Teacher Perceptions of Academic Focus, 2010-11 


Dedicated academic support is influenced by this 
year’s school-wide academic focus. 


Enrichment activities are influenced by this year’s 
school-wide academic focus. 

I use data specific to this year’s school-wide 
academic focus area to adjust instructional 
practices. 

I use data specific to this year’s school-wide 
academic focus area to monitor student progress. 


My instructional practice is influenced by this year’s 
school-wide academic focus. 


8 

35 

40 
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22 

35 

25 

19 


10 

34 
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0 


10 

35 

52 

0 
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0 20 40 60 80 100 


□ Not at all accurate □ Somewhat accurate □ Very accurate □ Don't know/NA 


EXHIBIT READS: In spring 2011, across all ELT schools, 40 percent of teachers reported that the statement, “Dedicated 
academic support is influenced by this year’s school- wide academic focus,” was very accurate. 

Source: Abt Associates’ Sun’eys of MA ELT School Teachers, Spring 2011, Item 20a through 20e. 

Sample: 918 teachers from ELT schools. Nonresponse rates across items ranged from 5.8 to 20.6 percent. 


Enrichment 

Findings 

• Most ELT schools have implemented separate enrichment classes. Nearly all students participated 
in enrichment classes/instruction, though the amount of time varied. Some schools also embedded 
enrichment activities within core classes. 

• The amount of time a typical student spent in enrichment varied considerably, from daily to 
weekly. Similar to last year, middle school students appeared to spend more time in enrichment 
than elementary students. 

• Approximately half of all ELT teachers reported that they taught at least one enrichment activity. 
Middle school teachers taught enrichment more often than elementary teachers. 

• Regular teachers/staff taught some enrichment activities at most ELT schools, and over half the 
ELT schools relied on partner organizations to provide some enrichment; of those latter schools, 
regular meetings were scheduled for partner staff and teachers to collaborate, an increase over 
reported efforts to integrate partners from the previous year. 
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• Most teachers reported that they and their students had some choice about selecting enrichment 
activities. The vast majority of teachers reported that all students had access to enrichment 
activities, and enrichment activities were of high quality. 

Context and Definition 

ELT schools are expected to add time to their schedules for enrichment both within core academic classes 
and/or in specialty classes. Enrichment opportunities should “connect to state standards, build student 
skills and interests, and deepen student engagement in school/learning in support of school-wide 
achievement goals.” 54 Separate enrichment classes, sometimes referred to as “electives,” are activities 
with either academic or non-academic content (e.g., yoga, robotics, math club, newspaper, or sewing) that 
only some students attend. All students might attend an enrichment class, although not all students 
necessarily attend the same [enrichment] activity or class. Using this definition, 17 principals reported 
having separate enrichment classes or electives. Five schools also had “embedded enrichment” or 
enrichment within core academic classes, in addition to enrichment classes or electives. 

Time 

According to principals, time for enrichment at ELT schools during 2010-11 ranged from 0 to 340 
minutes (or 5 hours and 40 minutes) per week, and schools allocated approximately 2 hours and 20 
minutes each week (139 minutes) for a typical student, on average. More time was allocated to 
enrichment each week for 8 th than 5 th grade (3 hours and 8 minutes and 1 hour and 48 minutes, 
respectively (not shown). 

The data described above represent aggregated findings across all ELT schools; the displays below (see 
Exhibit 4.16) indicate considerable school-level variation in time allocated to enrichment for students 
both within and across grade level. 


54 ELT Expectations for Implementation, Mass 2020 website: http://www.mass2020.org/node/14 Retrieved 
12/12/11. The Expectations were developed by ESE, Mass 2020, and Focus on Results. 
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Exhibit 4.1 6: Amount of Time Allocated to Stand-Alone Enrichment in ELT Schools Per Week, by 
School and Grade, 2010-11 



ELT 8th 



EXHIBIT READS: In spring 2011, School F allocated 0 hours per week for enrichment time for 5th graders. 

Notes: * Averages are calculated only among schools providing enrichment. 

School IDs have been randomly assigned in order to present school-level implementation data anonymously. 

Schools are presented in ascending order by amount of time allocated to enrichment. 

Source: Abt Associates' Interviews of MA ELT School Principals, Spring 2011, Item 11. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 l-8 lh grades for a total of 23 data points). Time data analyses 
exclude one ELT schools without either 5 th or 8 lh grades. 
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The frequency of enrichment ranged from daily to weekly. Even within a single school, the amount of 
time a student spent in enrichment sometimes depended on academic performance, as schools tried to 
balance the need for enrichment with the need for academic support. Ten principals reported that all 
students spent the identical amount of time in enrichment, whereas four indicated that students who 
needed more academic help or remediation had correspondingly less time for enrichment. Seven reported 
that students had weekly enrichment activities, six principals reported that students participated in 
enrichment two or three times a week, and in four schools, students had enrichment nearly every day. 

Approximately one quarter of teachers reported that the amount of time a typical student spent in 
enrichment each week was one block, two , or three blocks per week (29, 25, and 24 percent, 
respectively) (see Exhibit 4.17). Elementary teachers more often reported one block per week (45 
compared to 10 percent of middle school teachers). 


Exhibit 4.17: Teachers’ Reports about Students’ Time in Enrichment, By School Configuration, 
2010-11 



Percent of Teachers Reporting 


Overall 

Elementary 

Middle 

One time block of at least 45 minutes 

29% 

45% 

10% 

Two time blocks of at least 45 minutes each 

25 

17 

34 

Three or more time blocks of at least 45 minutes each 

24 

15 

35 

It is difficult to say because the amount of time varies widely 
across students 

16 

13 

19 

No Time 

7 

10 

3 


EXHIBIT READS: In spring 2011, across all ELT schools, 29 percent of teachers reported that a typical student in their 
class spent one time block of at least 45 minutes in enrichment activities each week. 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 13. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 7.0 percent. 
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The most popular time for enrichment was at the 
end of the day, especially at elementary schools, 
although some schools offered enrichment 
activities throughout the day. According to 
principals, 13 schools had some enrichment at the 
end of the day, and 12 schools offered enrichment 
during either the middle or first part of the school 
day (7 and 5, respectively). The time of day for 
enrichment in terms of scheduling generally 
reflected either grade-level or staffing (i.e., 
teacher- or partner-led) considerations. There are 
no clear patterns of timing for enrichment 
activities across school configurations. 

Decision-Making about Enrichment Activities 

The process of determining enrichment options 
varied by school. At about half of the schools 
(10), teachers presented ideas for enrichment 
classes, through submission of proposals to the 
principal to create courses aligned with their 
personal interests. Principals mentioned that 
course options were based at least partially on 
student interest at five schools. The majority of 
teachers reported that teachers have some choice 
about which enrichment activities they teach (86 
percent). 

Thirteen principals indicated that student 
preference was a major factor in determining 
students’ entry into specific enrichment activities. 

For example, students were sometimes asked to 
indicate their top three choices for enrichment. 

The majority of teachers (69 percent) agreed that 
students have some choice about which 
enrichment activities they take, though the level 
of agreement varied across schools (See Exhibit 
4.18). Overall, teacher reports indicated more 
student choice in 2010-1 1 than in the previous 
year (an increase of 15 percentage points). More 
middle school teachers reported student choice than elementary teachers (78 compared to 62 percent). 
Across all ELT schools, about one-third of students reported that they were able to choose some of their 


Enrichment at Matched Comparison Schools 

In 201 0-201 1 , seven of 1 9 comparison schools had 
implemented separate enrichment activities or electives 
into their school day. In addition, one school had 
embedded enrichment within core academic classes but 
did not offer electives. Examples of enrichment classes 
included chorus, band, drama, visual arts, science club, 
and bike repair. Enrichment classes were most often 
taught by regular classroom teachers (in five of seven 
schools). One school used paraprofessionals, and one 
used staff from partner organizations to teach some 
enrichment classes. 54 

Some students at comparison schools never had 
enrichment, while some had it every day. Three 
comparison schools offered enrichment classes twice a 
week, two offered enrichment weekly, and one offered 
daily enrichment activities to students. 55 The end of the 
day was the most common time for enrichment to be 
scheduled. 

At four comparison schools, all students participated in 
enrichment whereas at two schools, only some students 
participated. 56 Six principals mentioned student choice as 
a factor when assigning enrichment classes. The student 
to teacher ratio for enrichment classes was at or above 
20 : 1 . 

All comparison schools also reported providing voluntary 
after-school opportunities for their students, either through 
Community partnerships or led by school staff. Efforts to 
integrate partners into the ongoing school day varied. Five 
principals reported that communication between teachers 
and partner staff was on an as-needed basis, four that 
they provided formal orientation for partners, and two 
each that they provided training or professional 
development to partner staff and/or that they had regular 
meetings for teachers and partner staff. 

Comparison schools used school and district resources 
(two and three schools respectively) as well as other 
funds such as private grant funding in order to 
compensate partner organizations. Six principals reported 
providing no compensation to some partners. 


55 These data were missing from two comparison schools with enrichment. 

36 This information was missing from one comparison school that offers enrichment activities. . 
57 This information was missing from one comparison school that offers enrichment activities. 
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classes at school, although over three times as many middle school students reported this than elementary 
students (51 and 16 percent). 58 


Exhibit 4.18: Teachers’ Reports about Student Choice in Enrichment, By School 



DGPMFNRHJ ALOKEBCQ I 


EXHIBIT READS: In spring 2011, 21 percent of teachers from School D agreed that “All students have some choice about 
which enrichment activities they take.” 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Item 14b. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 20.4 percent. 


Content of Enrichment 

Enrichment classes at ELT schools were both academic and non-academic. Academic offerings included 
courses such as science club, ocean life, robotics, math league, urban ecology, astronomy, engineering, 
and architecture. Non-academic enrichment classes included arts-based activities such as knitting, drama, 
dance, puppetry, African drumming, woodworking, mask making, rock band, and origami. Physical 
activities were also offered such as swimming, basketball, lacrosse, soccer, and wellness/fitness. 

Some non-academic enrichment courses were deliberately structured to make connections with the 
academic frameworks. The majority of teachers reported that enrichment was clearly connected to grade 
level standards and/or curriculum frameworks (79 percent). Elementary teachers reported this more often 
than middle school teachers (87 compared to 68 percent). 

Staffing 

At many ELT schools, a mix of different types of staff taught enrichment, as in previous years. Nearly all 
of principals (16) reported that regular classroom teachers/staff taught some enrichment. About half of 
schools also relied on partner organizations for enrichment (10 schools). Some schools relied upon such 


58 

Note: This includes all classes and is not limited to enrichment. 
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staff as subject specialists, special education teachers, math or ELA coaches, or paraprofessionals to teach 
enrichment. At nine middle schools and seven elementary schools, classroom teachers taught at least 
some enrichment. In fewer schools (six middle and four elementary) teachers reported that they worked 
with partner staff. 

Similar to last year, just over half of all ELT teachers reported that they taught or co-taught one or more 
enrichment activities (53 percent). More middle school teachers reported that they taught enrichment than 
elementary teachers (59 compared to 49 percent). 

Participating Students 

All students participated in enrichment at 15 of the 17 schools, according to principals. At one school, it 
was nearly all students (98 percent of students), and at another school, it was most students (85 percent). 
The majority of teachers (85 percent) agreed that all students have access to enrichment activities, though 
the level of agreement varied across individual schools (See Exhibit 4.19). 

Enrichment classes were often smaller than core academic classes. In most schools, the student to teacher 
ratio for enrichment was 15:1 or smaller. Most teachers reported that the number of students in 
enrichment activities was conducive to effective instruction (78 percent), representing an increase of 22 
percentage points from the previous year. 


Exhibit 4.19: Teachers’ Reports about Student Access to Enrichment, By School 
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EXHIBIT READS: In spring 2011, 50 percent of teachers from School M agreed that “All students have access to enrichment 
activities.” 

Source: Abt Associates' Surveys ofMA ELT School Teachers, Spring 2011, Item 14a. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 18.0 percent. 
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Partnership Integration and Compensation 

During the 2010-11 school year, most ELT schools worked with partner organizations to provide 
instruction and/or additional learning opportunities during the school day. Twelve schools reported that 
they held meetings between regular classroom teachers and partner staff for sharing and collaboration to 
help integrate partners into the ongoing school day, and 1 1 schools reported that they held a formal 
orientation for partner organizations. Other schools provided professional development and/or training to 
partner staff and used communication on an as-needed basis between partners and classroom teachers 
(eight schools each). More middle schools provided professional development than elementary schools 
(six middle and two elementary schools). Overall, schools appear to have increased efforts to integrate 
partners in 2010-11 as compared to the previous year. 

Schools relied upon different resources to compensate partner organizations, including school-specific 
resources and additional district resources (nine and five schools, respectively). Other funding sources 
named by schools included the ELT grant, Title I funding, fundraisers/parent-teacher organizations, and 
other grants. The fact that the per-pupil grant amount for ELT was unchanged over several years meant 
that schools reported needing creative solutions for funding enrichment activities. Schools varied in which 
funding sources were used to compensate outside organizations according to the specific partner 
organizations involved, and two schools reported that they provided no compensation to at least some of 
their partners. 

Teacher Perceptions of Enrichment 

Overall, teachers reported that high quality enrichment was a valuable part of their extended school day, 
and most teachers responded positively about enrichment at their school (see Exhibit 4.20). The majority 
of teachers reported that most enrichment activities were of high quality (81 percent). Most teachers also 
agreed that enrichment activities were well integrated into the school schedule (79 percent) and occurred 
frequently enough to be valuable (76 percent). Middle school teachers reported that enrichment was 
frequent enough to be valuable more often than elementary teachers, which is consistent with the greater 
amount of time spent in enrichment by middle school students. Nearly three-quarters of all teachers 
agreed that the way in which students were grouped in enrichment was appropriate (73 percent), with 
elementary teachers reporting this more often (80 compared to 65 percent of middle school teachers). 
Overall, this was a 21 percentage point increase from the previous year. The vast majority of teachers also 
reported that most enrichment activities were taught by teachers/instructional staff in their school (88 
percent). 

Fewer than half of teachers reported that student behavior was problematic during enrichment (46 
percent). Similar to last year, middle school teachers encountered more behavior problems than 
elementary teachers (57 compared to 37 percent). At most schools, student performance in enrichment 
was not assessed similarly to core academic classes; systematic assessment of students in enrichment 
classes seems to be a continued challenge. Across ELT schools, only 45 percent of teachers were satisfied 
with the time available for students to pursue topics of interest to them. 
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Exhibit 4.20: Teacher Perceptions of Enrichment, By School Configuration, 2010-11 


Most enrichment activities are taught by teachers/instructional 
staff in this school. 

Teachers have some choice about which enrichment activities 
they teach. 

All students have access to enrichment activities. 


Enrichment activities are of high quality. 

Enrichment activities are well integrated into the school 
schedule. 

Enrichment activities are clearly connected to grade level 
standards and/or curriculum frameworks. 

The number of students in enrichment activities is conducive to 
effective instruction. 

Enrichment activities occur frequently enough to be valuable. 

The way in which students are grouped in enrichment activities 
is appropriate. 

All students have some choice about which enrichment 
activities they take. 

Student performance in enrichment activities is assessed 
similarly to assessments in core academic classes. 

Student behavior is problematic during enrichment activities. 

Teacher satisfied with time available for students to pursue 
topics of interest to them. 

Most enrichment activities are taught by community partner 
staff. 



% Agree or Strongly Agree 


EXHIBIT READS: In spring 2011, across all ELT schools, 88 percent of teachers agreed or strongly agreed that most 
enrichment activities were taught by teachers/staff in the school. 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Item 14a through 14m and Item lOe. 
Sample: 918 teachers from ELT schools. Nonresponse rates across items ranged from 10.1 to 24.3 percent. 
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Teacher Leadership and Collaboration 

Key Findings 

• More than half of ELT teachers (65 percent) participated in collaborative planning time weekly or 
more often. More than three-quarters of teachers participated monthly or more often (85 percent) 

• Only a small proportion of teachers (16 percent) reported never having participated in 
collaborative planning. 

• Teachers reported participating in multiple activities during collaborative planning time, 
including analyzing student data, strategizing about instructional practices, and/or reviewing 
student work. The majority of teachers who reported participating in an activity also reported that 
the activity was useful. 

• Teacher perceptions of principal leadership varied across schools. Teachers were more 
consistently positive about principals’ ELT-focused leadership than they were about principals’ 
leadership in general. 

Context 

One guiding principle of ELT is using time to broaden opportunities for teacher planning and preparation 
as well as school leadership (Mass 2020, 2011). ELT schools are expected to build a professional culture 
of teacher leadership and collaboration focused on improving instructional practice and meeting school- 
wide goals. Teacher surveys and principal interviews focused on participation in and perspectives about 
collaborative planning time, professional development, and leadership at the district, school, and 
individual level. 

Principals reported that collaborative planning time at their schools followed specific structures, including 
using designated facilitators, pre-articulated or approved agendas, and pathways for communicating the 
meeting’s content to administrators and others not in attendance. One school, for example, had a 
permanent elected facilitator position, while teachers rotated as facilitators at another school. Several 
principals described sharing documentation to communicate about collaborative planning meetings, while 
others used in-person debriefings. This section describes the frequency, organization, and focus of 
collaborative planning time in ELT schools. 

Collaborative Planning Time 

Time/F requency 

Overall, the large majority of teachers (84 percent) reported participating in collaborative planning at least 
monthly (see Exhibit 4.21). Less than one-fifth of teachers (16 percent) reported never having participated 
in collaborative planning. More middle school teachers reported that such meetings occurred at least once 
a week than did elementary teachers, on average (45 and 30 percent, respectively) 
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Exhibit 4.21: Frequency of Teacher Participation in Collaborative Planning Time, By School 
Type, 2010-11 


Overall 


Elementary 


Middle 



% Agree 


□ More than once a week nOnce a week □Twice a month □Monthly nNever 


EXHIBIT READS: In spring 2011, across all ELT schools, 37 percent of teachers reported that they participated in 
collaborative planning time more than once a week. 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Item 23. 

Sample: 918 teachers from ELT schools. Nonresponse rate for this item was 3.7 percent. 


Exhibit 4.22 presents information on how much time teachers reported spending in each collaborative 
planning time meeting overall and by school configuration. Each vertical line represents the full range of 
minutes reported by teachers in each respective group, and the box in the middle of the line represents the 
middle 50 percent of the distribution. The top of the box represents the 75 th percentile, and the bottom line 
of the box represents the 25 th percentile of the distribution. Within each box, both the median and the 
mean number of minutes for collaborative planning time are indicated. 

Teachers reported spending from 0 to 200 hundred minutes in each collaborative planning meeting across 
ELT schools. This range is large in part because the frequency of collaborative planning varies, so for 
example it is likely that the longer meeting blocks occur less frequently (e.g., three hours once per 
month). As Exhibit 4.25 shows, middle schools’ collaborative planning meetings lasted between 45 and 
60 minutes, and elementary level meetings were generally between 30 and 45 minutes long. ELT teachers 
reported spending an average of 50 minutes in each collaborative planning meeting. On a weekly basis, 
ELT teachers reported spending an average of an hour and a half collaboratively planning. 
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Exhibit 4.22: Range of Minutes Spent in Collaborative Planning Time Meetings, By School Type, 
2010-2011 
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EXHIBIT READS: The figure presents the range of the amount of time, per week, that teachers reported spending in 
collaborative planning meetings. The shaded box represents the middle 50 percent — from the 25 th to the 75 th percentiles — of the 
distribution of school day length, and the lines to the bottom and top represent the bottom and top quartiles, respectively, 
extending to the minimum and maximum values. The double thick line at the edge of the shaded box represents the median 
value for each distribution aligning with either the 25 lh or 75 th percentile. The diamond shape represents the mean value. The 
median value for collaborative planning meetings across all ELT schools is about 45 minutes, and the mean value is 50 minutes 
per week. 

Source: Abt Associates’ Interviews of MA ELT School Principals, Spring 2011, Item 22c. 

Sample: Time data analyses include 17 ELT schools (12-5 th and 1 l-8 lh grades for a total of 23 data points). Time data analyses 
exclude one ELT schools. 


Content and Structure 

The most frequent uses of collaborative planning time reported by teachers included analyzing assessment 
data for students in their classes (66 percent); strategizing about effective instructional practices and/or 
assessments (65 percent); and/or reviewing student work for students they teach (61 percent). Fewer 
teachers reported using collaborative planning time to participate in peer-to-peer mentoring (26 percent); 
perform administrative tasks (26 percent); and/or participate in coaching (24 percent). 
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As Exhibit 4.23 shows, generally, a majority of teachers who reported having participated in an activity 
during collaborative planning time also indicated that the activity was useful to them; about 70 percent 
(or more) of the teachers who participated in an activity also found it helpful. 


Exhibit 4.23: Participation in and Usefulness of Collaborative Planning Activities, 2010-2011 


Analyzing student assessment data for students in my 
classes 

Strategizing about effective instructional practices 
and/or assessments 

Reviewing student work for students I teach 
Discussing individual student needs for students I teach 

Planning lessons and instruction for my class(es) 

Discussing behavior management strategies for 
students in my classes 

Coordinating my instruction with special education staff 
for students I teach 

Coordinating my instruction with academic support 
personnel for students I teach 

Participating in other school-based professional 

Coordinating my instruction with ELL staff for students I 
teach 


] 52 


] 44 


] 46 


] 47 


] 37 
H 42 



] 25 


■ 66 
I 62 
I 62 
59 



35 


Performing administrative functions 
Participating in peer-to-peer mentoring 
Participating in coaching from school or district coach 

Other 



■ % Participated 
□ % Found useful 


0 20 40 60 80 100 


EXHIBIT READS: In spring 2011, across all ELT schools, 66 percent of teachers reported participating in analyzing student 
assessment data for students in their classes. 5 1 percent of the teachers who participated found the activity to be useful. 

Source: Abt Associates’ Sun’eys ofMA ELT School Teachers, Spring 2011, Item 25. 

Sample: 918 teachers from ELT schools. The binary nature of the survey items does not allow for the reporting of nonresponse 
rates. 


Over three-quarters of teachers reported that collaborative planning meetings had a structured format. 
Collaborative planning time meetings often had designated facilitators, set agendas, and mechanisms for 
informing the principal or coaches of what happened at these meetings such as logs, minutes, or debrief 
meetings. Additionally, at 10 of the 18 ELT schools, the Instructional Leadership Team (ILT) was 
involved in collaborative planning time because they intentionally shared staff members. For example, 
one principal explained that academic coaches on the ILT were also responsible for running collaborative 
planning meetings. At another school, one representative from each grade level on the ILT communicated 
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about ILT initiatives during collaborative planning meetings and raised staff issues from collaborative 
planning to the ILT. 

Overall, elementary school teachers reported somewhat higher levels of satisfaction with collaborative 
planning time than middle school teachers (62 percent and 52 percent, respectively; not shown). 
Additionally, 14 percent more elementary teachers than middle school teachers agreed that meetings were 
used in ways that contributed to the development of a professional learning community of teachers (84 
and 70 percent, respectively; not shown). 

How teachers shared and discussed instructional practices and how they used instructional strategies 
across subject areas varied across schools (Exhibit 4.24). At one school, for example, every teacher 
reported that teachers at their school shared and discussed instructional practices and that they used 
common instructional strategies across subject areas. At another school, about half the teachers reported 
such sharing and use of common instructional practices. Across ELT schools, the proportion of teachers 
who reported sharing and discussing instructional practices at their school ranged from 40 to 100 percent; 
and the proportion reporting that teachers used common instructional strategies across subject areas 
ranged from 62 to 100 percent. Typically, by school, a similar proportion of teachers agreed that teachers 
both shared and discussed instructional practices and that they used common instructional strategies. 


Exhibit 4.24: ELT Schools' Variation in Teachers’ Perception of Collaboration (ELT Schools Only) 
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EXHIBIT READS: In spring 2011, 40 percent of teachers from School L agreed that “Teachers at this school share and 
discuss instructional practices.” 

Source: Abt Associates’ Sun’eys of MA ELT School Teachers, Spring 2011, Item 7b 
Sample: 918 teachers from ELT schools. Nonresponse rates ranged from 5.7 to 9.7 percent. 
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Exhibit 4.24 (cont’d): ELT Schools' Variation in Teachers’ Perception of Collaboration (ELT 
Schools Only) 



EXHIBIT READS: In spring 2011, 62 percent of teachers from School H agreed that “Teachers at this school use common 
instructional strategies across subject areas.” 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, item 7c. 

Sample: 918 teachers from ELT schools. Nonresponse rates ranged from 5.7 to 9.7 percent. 


Professional Development 

Time and Frequency 

Teacher engagement in professional development and its contribution to a school-wide learning 
community at ELT schools was widespread. The distribution of teachers’ reported hours of professional 
development participation did not vary by school configuration; generally, about 85 percent of teachers 
participated in at least 10 hours of professional development over the course of the year. Overall, half of 
teachers reported having completed more than 26 hours of professional development over the course of 
the school year, 12 percent reported completing between 51 and 75 hours of professional development, 1 1 
percent completed more than 75 hours of professional development. The majority of ELT teachers 
participated in between 10 and 50 hours of professional development during 2010-11. More than three- 
quarters of ELT teachers (77 percent) agreed that in-school professional development contributed to the 
development of a professional learning community of teachers. 

Content and Structure 

According to principals, ELT schools have considerable autonomy over the content and structure of 
professional development. Across ELT schools, principals indicated that professional development was 
led by someone on their school staff: themselves, the ILT, the redesign or subject area coach, a teacher, or 
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some combination. Two principals noted that some of their professional development days were 
organized and led by the district, and three noted that professional development was sometimes led by a 
guest facilitator. At seven ELT schools, principals indicated that school partners participated in 
professional development meetings, and at an additional school, this kind of integration was in the 
planning stage. Interestingly, different schools that partnered with the same local organization did not 
necessarily follow the same policies regarding including the partner organization in professional 
development. 

School staff reported that they usually chose the topic and developed the agenda for professional 
development meetings. Principals noted a range of key topics covered by professional development, 
including focusing on data, instructional strategies, writing, and bullying. 

Principals at 17 of the 18 ELT schools reported visiting classrooms in their schools at least weekly. The 
purposes of these visits varied, from formal teacher evaluations to general assessments of classroom 
climate and atmosphere. At seven schools, principals used classroom visits to ensure that teachers were 
using common instructional strategies, focusing on particular curricula, and teaching lessons that were 
adequately rigorous. 

Observational visits by teachers to other teachers’ classrooms occurred less frequently. At 16 schools, 
these visits occurred on a monthly basis or less. Seven principals indicated that they hoped peer 
observations would happen more often in the future. 

According to survey data, teachers accessed a variety of professional development opportunities over the 
course of the year. The most frequent professional development activities included: attending out-of- 
school professional development (53 percent), peer modeling (49 percent), and academic coaching from 
in-school coaches (48 percent). Figure 4.25 shows few differences in participation between middle 
schools and elementary schools, except that academic coaching was slightly more common at the 
elementary school level than in middle schools (53 and 41 percent of teachers, respectively, reported 
participation). 
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Exhibit 4.25: Sources Teachers Use to Support Instruction, by School Type, 2010-11 (ELT 
Schools Only) 



Percent Used 


Overall 

Elementary 

Middle 

Out-of-school professional development 

53% 

52% 

56% 

Peer support (e.g., peer coaching, peer modeling, peer 
demonstration lessons) 

49 

47 

52 

Academic coaches in school 

48 

53 

41 

Principal and/or other administrative support 
(e.g., coaching, modeling) 

27 

28 

26 

Special education aide 

27 

27 

27 

Partner staff 

26 

27 

24 

Technical assistance from outside district 

9 

8 

11 

Other 

3 

3 

3 


EXHIBIT READS: In spring 201 1, across all ELT schools, 53 percent of teachers reported using out-of-school professional 
development to support instruction during the 2010-11 school year. 

Source: Abl Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 28. 

Sample: 918 teachers from ELT schools. The binary nature of the survey items does not allow for the reporting of 
nonresponse rates. 


Satisfaction 

Overall, nearly 80 percent of teachers agreed that in-school professional development activities 
contributed to the development of a professional learning community of teachers, and , the majority of 
teachers reported being satisfied with the amount of time allotted for professional development (66 
percent overall). Less than half (40 percent) of teachers surveyed reported being unable to attend the 
professional development opportunities they wanted to attend; this proportion did not vary by school type. 
Teacher satisfaction with the amount of time for professional development did not appear to vary by 
school type (not shown). 


Leadership 

Teacher Perceptions of District Leadership 

Teachers’ perceptions about the quality of general district leadership, including communication, 
management, and guidance varied, and were generally positive. Notably, 85 percent of teachers overall 
agreed that the district leadership was supportive of ELT. Over half of teachers overall agreed that district 
leadership was responsive to their school and teacher concerns (see Exhibit 4.26). 
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Exhibit 4.26: Teacher Attitudes about District Leadership, by School Type, 2010-11 11 (ELT 
Schools Only) 



Percent Agree 

The district leadership... 

Overall 

Elementary 

Middle 

actively supports ELT implementation at this school 

85% 

88% 

81% 

communicates a clear vision for our district's schools 

73 

74 

71 

is interested in the professional development of teachers at 
this school 

69 

71 

67 

effectively manages our schools 

62 

64 

58 

provides timely guidance on instructional practice, 
curriculum, etc. 

60 

66 

53 

is responsive to this school and teacher concerns 

51 

52 

51 


EXHIBIT READS: In spring 201 1, across all ELT schools, 85 percent of teachers reported that district leadership actively 
supports ELT implementation at their school. 

Source: Abt Associates’ Sun’eys ofMA ELT School Teachers, Spring 2011, Item 7cc through 7hh. 

Sample: 918 teachers from ELT schools. Nonresponse rates across items ranged from 13.7 to 21.1 percent. 


Teacher Perceptions of Principal Leadership 

Over two-thirds of teachers generally agreed that their principals possessed leadership capabilities, 
evidenced by instructional leadership, management skills, and providing opportunities for teacher 
leadership and recognition (74, 69, and 67 percent, respectively). Teachers’ perceptions of principal 
leadership were consistent across school type. 

Teacher perception of principal leadership varied by school and was generally positive. Interestingly, 
across schools, teachers were more consistently positive about their principal’s ELT leadership than they 
were about their principal’s leadership more generally. As Exhibit 4.27 below shows, across schools, 
teachers were generally in agreement about various aspects of ELT leadership; there was more variation 
in teacher perception of principals’ management, communication, and staff development skills. 

Within a school, from 39 to 99 percent of teachers agreed that their principals were effective managers, 
50 to 99 percent agreed that their principals communicated effectively, and 44 to 94 percent of teachers 
agreed that their principals monitored teacher progress and provided feedback. 
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Exhibit 4.27: Teachers’ Perceptions of Principal’s Leadership 
Panel 1: Teachers’ Perceptions of Principal’s ELT Leadership 


The principal and/or school leadership team are strong advocates of ELT. 



The principal and/or school leadership team communicate a clear vision of ELT for 
our school. 

100 



The principal and/or school leadership team monitor performance and progress of 
implementation of ELT in the school. 

100 



EXHIBIT READS (Panel 1): In spring 2011, 86 percent of teachers at School N reported that the principal and/or school 
leadership team were strong advocates of ELT. 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 7g, 71, 7m. 

Sample: 918 teachers from ELT schools. Nonresponse rates across items ranged from 3.7 to 8.8 percent. 


Abt Associates Inc. 


Chapter 4. Detailed Implementation Findings | pg. 89 


ELT Evaluation Year 5 Integrated Report 


Panel 2: Teachers’ Perceptions of Principal’s Leadership in General 

The principal and/or school leadership team are effective managers who make the 
school run smoothly. 



The principal and/or school leadership team communicate effectively with teachers 
about improving instructional practices at this school. 



The principal and/or school leadership team monitor and provide feedback to 
teachers about their performance. 



EXHIBIT READS (Panel 2): In spring 2011, 39 percent of teachers at School H reported that the principal and/or school 
leadership team were effective managers who made the school run smoothly. 

Source: Abt Associates’ Sun’eys of MA ELT School Teachers, Spring 2011, Item 7v, 7r, 7n,. 

Sample: 918 teachers from ELT schools. Nonresponse rates across items ranged from 3.7 to 8.8 percent. 
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Collaborative Planning Time and Professional 
Development at Matched Comparison Schools 

According to principals, at 18 of the 19 comparison schools, 
teachers also participated in collaborative planning time. .. 
Collaborative planning meetings appeared to have a distinct 
structure, with a facilitator, agenda, and method of 
communicating the meeting’s content to the school 
administration. At one school collaborative planning time was 
informal, in that teachers were welcome, but not required, to 
arrange meetings during their shared preparation time. 

Professional development in comparison schools was typically 
facilitated by some combination of the principal, teachers, ILT, 
and the academic coaches. All principals noted that topics and 
agenda for professional development were identified by the 
ILT, the coaches, or the principal, and often included input from 
the teachers. One principal noted that the district played a 
leadership role in professional development. When asked 
about the focus of professional development, there was not a 
dominant topic; principals reported that professional 
development topics at their schools ranged from MCAS 
preparation, to common instructional strategies, to writing and 
oral skills, to student data analysis, to school climate and 
bullying. 

Twelve principals at comparison schools noted that it was a 
priority for them to visit teachers’ classrooms at least weekly, 
some as frequently as every day. The purposes of these visits, 
according to principals, varied from formal teacher evaluations 
to understanding the student experience. Some principals 
reported that teachers visited one another’s classrooms in 
comparison schools 

According to principals, teachers also visited each other’s 
classrooms, though they reported that these visits were 
infrequent, informal, and/or by invitation. At three schools, 
principals mentioned structured “rounds” or learning walks that 
occurred a few times per year and required teachers to visit 
one another’s classrooms. At five schools, principals noted that 
peer observation did not happen formally or as often as they 
would like. 
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Chapter 5. Non-Academic Outcomes in ELT and Matched 
Comparison Schools 


This chapter examines the relationship between ELT and selected non-academic outcomes for schools, 
students and teachers, as outlined in the conceptual model and discussion of research questions in Chapter 
2. The ELT theory of change hypothesizes that as a result of the initiative, students and teachers will 
experience a number of positive outcomes, including more time for meaningful peer-to-peer and teacher- 
student interactions; greater student engagement and satisfaction due to increased enrichment 
opportunities; increased school leadership and district supports; and greater teacher satisfaction about 
instructional time due to adequate time to plan, prepare and instruct. This chapter explores each of these 
themes in turn, using data collected from semi-structured principal interviews, student and teacher 
surveys, 59 and extant data (publicly available school-level data and Student Management Information 
System (SIMS) data). Data from all three cohorts are presented together to understand the effects of ELT 
after at least three years of implementation. 

The chapter begins by describing the characteristics of the study schools by group status (ELT and 
matched comparison) using publicly available school-level data to provide context for the subsequent 
results. It then examines the allocation of instructional time at ELT schools using data from semi- 
structured principal interviews, followed by an assessment of non-instructional time using student survey 
data. Next, it examines how the expanded day may be related to teacher and student outcomes, focusing 
on teacher survey responses to questions about school and district leadership and teachers’ satisfaction 
with various aspects of teaching, and then focusing on findings about student and teacher relationships 
and student satisfaction and engagement using survey data and state-provided SIMS data. 


Current Characteristics of ELT and Matched Comparison Schools 

Exhibit 5.1 presents key characteristics of the ELT and matched comparison schools across cohorts in 
2010-11 to provide important context for outcome findings. Recall that the 2010-11 study sample 
includes 18 ELT schools and 19 matched comparison (MC) schools chosen based on their similarity to 
ELT schools prior to the start of ELT. The 2010-11 school year characteristics of these schools are 
presented below. 

Consistent with prior years’ findings, the majority of ELT schools have been identified for improvement, 
corrective action, or restructuring based on their ELA or math achievement scores over the previous two 
years. Further, the ELT schools tend to serve large proportions of low income and minority students. Note 
that characteristics of MC schools are generally similar to those of the ELT schools. 


59 Because the study team substantially revised the surveys before the 2010-11 administration, it was not possible 
to combine data from prior survey administrations in the current analyses. 
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Exhibit 5.1 : Current Characteristics of Sample Schools, by ELT Status 



ELT 

MC 

Grade Span 


Elementary school 

6 

6 

K-8 school 

4 

6 

Middle school 

8 

7 

School Location 


City 

9 

11 

Suburb 

6 

7 

Town 

2 

1 

Rural 

1 

0 

School Size 


600 students or more 

7 

7 

400-599 students 

8 

9 

200-399 students 

2 

3 

Fewer than 200 students 

1 

0 

Low Income Student Population 


75 percent or more 

9 

8 

50-74 percent 

9 

9 

Less than 50 percent 

0 

2 

Minority Student Population 


75 percent or more 

7 

4 

50-74 percent 

5 

9 

25-49 percent 

4 

5 

Less than 25 percent 

2 

1 

SPED Student Population 


20 percent or more 

7 

11 

10-19 percent 

10 

8 

Less than 1 0 percent 

1 

0 

LEP Student Population 


20 percent or more 

7 

4 

10-19 percent 

3 

6 

Less than 1 0 percent 

8 

9 

Met Aggregate Adequate Yearly Progress (AYP) in 2011 


English language arts 

7 

4 

Math 

2 

6 

ELA Accountability Status in 201 1 


No status (AYP met for previous two years) 

2 

4 

Identified for improvement, corrective action, or restructuring 

16 

15 

Math Accountability Status in 2011 


No status (AYP met for previous two years) 

1 

4 

Identified for improvement, corrective action, or restructuring 

17 

15 


EXHIBIT READS: In the 2010-11 school year, the study sample includes 6 ELT elementary schools and 6 matched 
comparison elementary schools. 

Source: School-level data tables downloaded from the MA ESE website (http://profiles.doe.mass.edu/). 

Sample: Eighteen ELT and 19 matched comparison schools. 
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Comparing Time in ELT and MC Schools 

Key Findings 

• The length of the ELT school day was significantly longer for 5 th and 8 th grade students than 
would be expected in the absence of ELT. 

• ELT schools allocated significantly more time for ELA, math, and science classes for 5 th and 8 th 
graders than would be expected in the absence of ELT. 

• ELT schools allocated significantly more time for non-core classes and specials for 5 th grade 
students than would be expected in the absence of ELT. 

• ELT schools allocated significantly more time for enrichment activities for 5 th and 8 th grade 
students than would be expected in the absence of ELT. 

• ELT schools allocated significantly more time for transitions, recess, snack, lunch, and 
homeroom for 5 th and 8 th grade students than would be expected in the absence of ELT. 

• A statistically smaller proportion of students in ELT schools reported that they attend an 
academic club than would be reported in the absence of ELT. 

• Significantly fewer students in ELT schools attended an after-school program than would be the 
expected in the absence of ELT. 

Instructional Time 

Principal responses from semi-structured interviews conducted in the spring of 201 1 are used to compare 
school-level averages of time allocated to various instructional activities in ELT and matched comparison 
schools. Differences in school-level averages are assessed using models that regress instructional time on 
an ELT indicator and dummy variables for each pair of matched schools. Because study schools allocated 
different amounts of time for particular grades, models are run separately for 5 th and 8 th grade students. 

Exhibit 5.2 displays the actual average number of hours and minutes allocated for various activities in 
ELT and matched comparison schools, as well as the difference in means and the results of the regression 
(discussed above) that assesses differences in school-level averages. In addition to the length of the school 
day, differences in the time allocated to the following activities were examined: 

• Core academic subjects: courses focused on four core subject areas (English or ELA, math, 
science, and social studies) that all students are required to take; 

• Non-core classes and specials: courses other than the four core subjects that all students in a 
school or grade are required to take (e.g., foreign languages, philosophy, physical education (PE) 
or gym, art, music); 

• Academic support period: an extra block of academic time that targets specific academic skills 
according to student needs/skills. Is generally not the choice of students (e.g., MCAS prep, 
academic leagues); 

• Enrichment: courses/activities with either academic or non-academic content (e.g. yoga, robotics, 
math club, newspaper, sewing) that only some students attend. All students might attend an 
elective but not all attend the same elective; 

• Transitions: lunch, recess, snack, transitions, and; 

• Homeroom: homeroom/advisory 
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Exhibit 5.2: Differences between ELT and Matched Comparison Schools in Hours and Minutes 
Allocated to Various Activities, 2010-11 



Actual 

ELT 

School 

Mean 

Actual 
Matched 
Comparison 
School Mean 

Difference in 
Means 

Statistical 

Significance 

(p-value) 

Grade 5 

Length of school day 

7:42 

6:04 

1:38 *** 

<.0001 

All Core Academic Subjects 

4:55 

4:15 

0:40 *** 

<.0001 

ELA 

2:05 

1:50 

0:15 ** 

0.009 

Math 

1:28 

1:13 

0:15 *** 

0.001 

Science 

0:45 

0:34 

0:11 * 

0.036 

Social Studies 

0:37 

0:37 

0:00 

0.967 

Non-Core and Specials 

1:01 

0:44 

0:17 *** 

<.0001 

Academic Support Time 

0:21 

0:11 

0:10 * 

0.030 

Enrichment Time 

0:20 

0:04 

0:16 *** 

<.0001 

Transitions Time 

0:49 

0:42 

0:07 *** 

0.001 

Homeroom 

0:22 

0:11 

0:11 *** 

<.0001 


Grade 8 


Length of school day 

7:54 

6:21 

1:32 *** 

<.0001 

All Core Academic Subjects 

4:37 

3:43 

0:53 *** 

<.0001 

ELA 

1:23 

1:01 

0:22 *** 

<.0001 

Math 

1:16 

0:59 

0:17 *** 

<.0001 

Science 

1:05 

0:52 

0:13 *** 

<.0001 

Social Studies 

0:52 

0:51 

0:01 

0.589 

Non-Core and Specials 

1:05 

1:04 

0:02 

0.748 

Academic Support Time 

0:34 

0:32 

0:02 

0.834 

Enrichment Time 

0:31 

0:14 

0:17 *** 

<.0001 

Transitions Time 

0:47 

0:36 

0:11 *** 

<.0001 

Homeroom 

0:20 

0:13 

0:08 *** 

<.0001 


EXHIBIT READS: ELT schools serving 5 th grade students were in session for an average of about 7 hours and 42 minutes per 
day, compared to 6 hours and 4 minutes per day in matched comparison schools serving 5 th grade students. The difference 
between the two groups was statistically significant. 

Notes: 

* statistically significant at p < .05 level 

** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates' Interviews ofMA ELT and Matched Comparison School Principals, Spring 2011, Items 9d, 11. 

Sample: Grade 5: 12 ELT and 12 matched comparison schools. Grade 8: 1 1 ELT schools and 11 matched comparison schools. 


Results of models examining time allocation indicate that ELT schools were in session for over an hour- 
and-a-half per day longer than would be estimated in the absence ELT. This significantly greater length of 
the ELT school day translated into more hours allocated to almost all of the activities examined. 

However, the largest differences were observed in the amount of time ELT schools allocated to core 
academic classes. ELT schools allocated about three-quarters of an hour more for 5 th and 8 lh grade 
students in these classes than would have been the case absent ELT in either grade. 
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A deeper examination of time allotted to core academic classes suggests that schools may prioritize extra 
time in particular subjects according to grade. While ELT schools allocated most of their additional 5 th 
grade core academic time to Math and Science (15 more minutes for each than in the absence of ELT), 
ELT schools with 8 th graders focused most of their additional time for core academics on ELA (22 more 
minutes than the counterfactual). The time data provided do not indicate that ELT schools allocated more 
time to Social Studies, a subject not tested in MCAS examinations. 

The expanded time also led to increased time allocated for non-core classes and specials, enrichment, and 
other activities. Time in non-core classes and specials for 5 th graders in ELT schools was 17 minutes more 
than would be expected in the absence of ELT. Notably, schools allocated almost 20 more minutes to 
enrichment classes, a key component of ELT implementation, than would be expected without ELT. 
Finally, as would be expected given the increased length of day, ELT schools devote more time to lunch, 
recess, snack, transitions, and homeroom time, than would be expected in the absence of ELT. 

It should also be noted that although 8 th grade students in ELT schools had most of their expanded time 
allocated to core academic classes (53 minutes), time allocated to other activities was also substantial (40 
minutes). Conversely, as compared to the additional amount of time allocated to core academic classes 
(40 minutes), 5 th grade students in ELT schools spent more of their expanded day in other activities (61 
minutes). 

Non-lnstructional Time 

The underlying theoretical model of ELT posits that students are provided more stimulating opportunities 
during school; it also may hold that the expanded day allows for increased participation in structured 
activities, such as academic clubs, and that the longer day limits time available for unstructured leisure, 
such as hanging out or spending time at home with family. Exhibit 5.3 presents the results of students’ 
participation in structured activities, after-school activities, recreation and leisure activities, and time 
spent at home with friends and family — outcomes drawn from student surveys. 

The effect of ELT on non-instructional activities is assessed using multi-level models that combine 5 th 
and 8 th grade student survey responses 60 and estimate the overall effect of ELT on student outcomes after 
accounting for the clustering of students within schools and classrooms. These models include an ELT 
indicator, dummy variables for each pair of matched schools, and school and student-level 
demographics. 61 As described in Chapter 2, the study team used factor analyses to determine which 
survey items would be presented in this chapter (see Appendix E for a presentation of all survey items 
corresponding to the ELT outcomes examined in this chapter). Thus, items presented below and in 
subsequent exhibits that draw from survey items are selected outcomes. 


60 While there are differences between 5 th and 8 lh grade student responses, the effect of ELT does not differ across 
grades. Therefore, 5 th and 8 th grade student survey responses were combined to fit models estimating the overall 
effect of ELT on students. In addition, note that because only 5 lh and 8 th grade students were surveyed, the 
responses cannot be assumed to represent the entire student populations in the study schools. 

61 The school -level characteristics included the total enrollment, the percentage of students eligible for free and 
reduced priced lunch, and the percentage of minority students served by the school. Student characteristics 
included gender, and whether or not the student reported speaking a language other than English most of the 
time at home. 
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Results presented in Exhibit 5.3, as well as in all other exhibits in the chapter, present actual ELT means 
for a particular outcome, along with the estimated non-ELT mean, the estimated difference, and the 
statistical significance of the estimated difference. The non-ELT estimated mean is a regression-adjusted 
estimate of what would have happened in ELT schools if ELT had not been implemented. As a result, 
findings are described in terms of actual ELT outcomes relative to the estimated outcomes in the 
counterfactual. 


Exhibit 5.3: Participation in Non-lnstructional Activities, 2010-11 


Percent of 



Respondents 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


Participation in Structured Extracurricular Activities 


Attending an academic club (science club, math club, etc.) 
Participating in student government 
Working on a school newspaper or magazine 

12% 

11 

12 

17% 

13 

11 

-5.7* 

-1.9 

0.1 

0.020 

0.467 

0.940 

Participation in After-School Activities 



Do you go to any after-school activities this school year? 

39 

50 

-11.7 * 

0.025 

Participation in Recreational or Leisure Activities 



Spending time online using a computer 

80 

78 

2.6 

0.520 

Texting or talking to friends 

87 

85 

1.7 

0.521 

Spending time with friends 

90 

90 

-0.1 

0.972 

Time Spent At Home With Friends And Family 



Home alone or with younger siblings 

43 

39 

3.6 

0.316 

With your friends and no adults 

47 

49 

-1.4 

0.701 

With an older sibling or parent 

70 

69 

0.7 

0.818 


EXHIBIT READS: Twelve percent of students in ELT schools indicated that they participated in an academic club, compared 
to an estimated 17 percent in the absence of ELT. The difference between the two groups was statistically significant. 


Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Suiyeys of MA ELT and Matched Comparison School Students, Spring 2011, Items 12n, 12e, 12f. 13, 
12r, 12p, 12d, 15a, 15b, 15c. 

Sample: 4840 students. Nonresponse rates across ELT and matched comparison student survey items ranged from 3.2 to 21.1 
percent. 


Results of these models indicate that, despite the expanded length of the ELT school day, there were 
generally no differences between students’ participation in activities as a function of their schools’ ELT 
status. However, significantly fewer students in ELT schools indicated that they spent time before, during, 
or after-school in an academic club, compared to the counterfactual. Further, consistent with the finding 
on the expanded length of the ELT school day, fewer students in ELT schools reported that they attend an 
after-school program than would be estimated without ELT (39 percent versus 50 percent, respectively). 
Presumably, many of the activities students would attend outside of (non-ELT) school occur within the 
(ELT) school day. It is interesting to note that a higher percentage of students across ELT and MC 
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schools reported attending an after-school program compared to the student responses in 2009-10 (22 
percent and 42 percent, respectively). It is possible therefore, that students this year may have 
misinterpreted the question and reported that they attended an after-school program that was actually part 
of schools’ expanded time, rather than a stand-alone program occurring outside of the school day. 

The majority of students across all schools reported spending time on the computer, interacting with 
friends, or being at home with an older sibling. Fewer students reported spending time at home alone with 
friends or younger siblings, a finding which may reflect the respondents’ age. 

In sum, while students’ non-instructional time was generally not affected by expanded time, ELT students 
spent significantly more time in a variety of activities throughout their day, compared to what would be 
expected in the absence of the intervention. If implemented successfully, increases in the length of the 
school day may be related to a number positive student and teacher outcomes. These outcomes are 
reviewed in the following section, beginning with a discussion of teacher outcomes and concluding with 
an analysis of student outcomes. 


Comparing Teacher and Student Non-Academic Outcomes between ELT and non- 
ELT Schools 

Teacher Outcomes 

Key Findings 

• A significantly higher proportion of teachers in ELT schools reported that the length of the day 
allows them to accomplish their teaching goals and cover the amount of instructional material 
their students need to learn than would be expected in the absence of ELT. 

• A significantly higher proportion of teachers in ELT schools reported that they are satisfied with 
the amount of time available for instruction in ELA, math, and science than would be expected in 
the absence of ELT. 

• A significantly higher proportion of teachers in ELT schools reported that they are satisfied with 
the amount of time available for academic support, enrichment activities and for students to 
pursue topics of interest than would be expected in the absence of ELT. 

• Significantly more teachers in ELT schools reported that they are satisfied with the amount of 
time available for collaborative planning and that the length of the day allows for coordination of 
instruction than would be expected in the absence of ELT. Conversely, significantly fewer 
teachers in ELT schools than the counterfactual reported that the amount of collaborative 
planning time is a problem area. 

Perceptions of School and District Leadership 

A key component of ELT implementation is the expectation of a professional culture of leadership and 
collaboration. School and district leaders are expected to include teachers, students, families, partners, and 
other community stakeholders in the ELT implementation process to help meet school-wide achievement 
goals. With these expectations in mind, ELT should increase opportunities for staff at all levels to 
collaborate and interact with school and district leaders. As such, teachers in ELT schools may be 
hypothesized to be more likely to perceive that school and district leaders are effective managers, actively 
engaged, adaptable, receptive to feedback and supportive of staff growth and development than in the 
absence of ELT. 
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Similar to analyses run on student surveys, models assessing the relationship between ELT and teacher 
outcomes use multi-level models that adjust for the clustering of teachers within schools and include an 
ELT indicator, dummy variables for each pair of matched schools, and school and teacher-level 
demographics. 62 Results in exhibits present the actual ELT mean relative to the regression-adjusted non- 
ELT mean for selected outcomes. 

Some of the differences in teacher perceptions were fairly large, but none rose to the level of statistical 
significance because the minimal detectable effect size was even larger. Exhibit 5.4 presents the findings 
on teacher perceptions of school and district leadership. On the whole, there were no statistically 
significant differences in teachers’ perceptions of school and district leadership as a function of ELT 
status. Nevertheless, some results merit further consideration. For example, the majority of teachers in 
ELT and non-ELT schools indicated that the principal and/or school leadership team were actively 
engaged, effective instructional leaders who make the school run smoothly. In addition, teachers in both 
school types were generally satisfied with the principal and/or school leadership team’s ability to receive 
feedback, adapt school policies, and communication with staff. While perceptions of district leadership 
were slightly less positive, most teachers indicated that the district leadership effectively managed their 
(respective) school. Note that while some of the estimated differences in teacher perceptions were fairly 
large, none rose to the level of statistical significance because the minimal detectable effect size (MDES), 
a measure denoting how large changes in teacher outcomes must be in order to detect statistically 
significant differences, was larger than the estimated differences. Our calculations indicate that the MDES 
for the all the teacher outcomes tested (those reported both in this chapter and Appendix E) ranges from 4 
to 24 percent, meaning that estimated differences would have to be between 4 and 24 percent in order to 
be detected as statistically significant. Among the factors contributing to the wide range of MDES are: 
differences in response rates, intra-class correlations, and the amount of outcome variance explained by 
pair fixed effects and other covariates included in the models. 


62 The school-level characteristics included in the model measured total enrollment, the percentage of students 
eligible for free and reduced priced lunch, and the percentage of minority students served by the school during 
the 2010-201 1 school year. The teacher-level characteristics included in the model measured the total number of 
years the teacher reported teaching/working in a school, including the 2010-11 school year. 
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Exhibit 5.4: Teacher Perceptions of School and District Leadership 



Percent of 
Respondents 




Actual 
ELT Mean 

Estimated 

Non-ELT 

Mean 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Leadership 



The principal and/or school leadership team are effective 
managers who make the school run smoothly 

75% 

80% 

-4.4 

0.620 

The principal and/or school leadership team are 
instructional leaders in our school 

70 

72 

-2.4 

0.701 

Level of Adaptability 



The principal and/or school leadership team are actively 
engaged in school improvement initiatives 

90 

94 

-4.2 

0.343 

The principal and/or school leadership team adapt or 
refine school policies and/or practices as needed 

78 

82 

-3.9 

0.519 

The principal and/or school leadership team monitors 
progress of students 

90 

87 

3.3 

0.535 

Communication 



The principal and/or school leadership team are receptive 
to feedback on teacher morale/school culture 

68 

78 

-10.2 

0.229 

The principal and/or school leadership team provide 
feedback about teacher performance 

72 

80 

-8.1 

0.157 

The principal and/or school leadership team communicate 
effectively about improving instructional practices 

74 

76 

-1.6 

0.828 

Support for Staff Development 



The principal/school leadership team provide opportunities 
for staff leadership and/or recognition of staff successes 

67 

80 

-12.6 

0.136 

The principal/school leadership team encourages the 
development of a professional learning community 

85 

83 

2.3 

0.650 

The principal/school leadership team are supportive of 
staff growth and development 

86 

86 

0.3 

0.950 

Perceptions of District Leadership 



District leadership effectively manages our schools 

62 

56 

6.6 

0.396 

District leadership provides timely guidance on 
instructional practices 

61 

55 

6.0 

0.283 

District leadership is responsive to this school and teacher 
concerns 

52 

49 

2.6 

0.748 


EXHIBIT READS: Seventy-five percent of teachers in ELT schools reported that the principal and/or school leadership team 
were effective managers who made the school run smoothly, compared to an estimated 80 percent in the absence of ELT. The 
difference between the two groups was not statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Items 7g, 7k, 7h, 7i, 7n, 7j 7m, 71, 7p, 27e, 7o, 7cc, 
7ff, 7gg; Matched Comparison Teacher Surveys, Items 7e, 7i, 7f, 7g, 71, 7h, 7k, 7j, 7n, 26e, 7m, 7v, 7y, 7z- 
Sample: 1609 teachers. Nonresponse rates across ELT and matched comparison teacher survey items ranged from 5.8 to 20.7 
percent. 
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Satisfaction with Time Available for Instructional Activities, Planning and Professional Development 

One of the most important components of ELT is that it be used to broaden opportunities for teachers, 
particularly to improve instructional practice. Schools that successfully implement ELT should provide 
teachers with more time for instructional activities and more time to plan, collaborate, and participate in 
professional development opportunities. 

Time Available for Instructional Activities, Planning and Professional Development 

Exhibit 5.5 suggests that teachers in ELT schools were generally more satisfied with time available for 
instruction than would be the case in the absence of ELT. A significantly larger proportion of teachers in 
ELT schools reported that the length of their days allowed them to accomplish their teaching goals and 
cover the amount of instructional material their students needed to learn, than would have been expected 
in the absence of ELT. ELT teachers were similarly satisfied with the amount of time their schools 
allotted for various instructional activities. Ninety percent or more of teachers in ELT schools were 
satisfied with the amount of time available for instruction in math and ELA, compared to about three- 
quarters in the absence of ELT. Teachers in ELT schools were also more likely to report that they were 
satisfied with the amount of time available for instruction in science than were teachers in the comparison 
schools (81 and 54 percent, respectively). Teachers in ELT schools were also significantly more satisfied 
with the amount of time available for students to pursue topics of interest and with the time available for 
academic support and enrichment activities. Notably, while ELT teacher satisfaction with time available 
for instruction was significantly higher compared to what would be expected in the absence of ELT both 
this year and last year (and magnitudes of estimated differences are roughly equivalent), actual ELT 
percentages and estimated non-ELT percentages are higher this year than last. 

There was no statistically significant effect of ELT on teachers’ abilities to differentiate instruction for 
students of multiple abilities or to use a variety of instructional strategies. Over 90 percent of all teachers 
reported satisfaction in these areas. 
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Exhibit 5.5: Time Available for Instructional Activities 



Percent of 
Respondents 




Actual 
ELT Mean 

Estimated 

Non-ELT 

Mean 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Teacher Perceptions of Instructional Time 



Length of day allows me time to accomplish teaching 
goals 

80% 

66% 

13.7 * 

0.024 

This year, 1 am able to cover the amount of instructional 
material my students need to learn 

74 

63 

10.9 * 

0.028 

This year, 1 am able cover the amount of instructional 
material my school expects 

82 

75 

6.5 

0.261 

Time Available for Instructional Activities 



Time available for enrichment activities 

68 

39 

29.3 ** 

0.002 

Time available for instruction in science 

81 

54 

26.0 ** 

0.001 

Time available for dedicated academic support 

64 

45 

19.5 * 

0.020 

Time available for students to pursue topics of interest 

44 

28 

16.3 * 

0.015 

Time available for instruction in ELA 

89 

74 

15.5 * 

0.010 

Time available for instruction in social studies 

71 

56 

15.2 

0.060 

Time available for instruction in math 

92 

78 

13.6 ** 

0.008 

Time available for physical activity for students 

41 

42 

-1.7 

0.808 

Time Available for Differentiated Instruction 



This year, 1 am able to differentiate instruction for students 
of multiple abilities 

89 

86 

2.9 

0.311 

This year, 1 am able to use a variety of instructional 
strategies (e.g., project-based learning, small-group 
learning) to teach my students 

92 

91 

1.0 

0.692 


EXHIBIT READS: Eighty percent of teachers in ELT schools indicated that the length of the school day allowed them to 
accomplish their teaching goals, compared to an estimated 66 percent in the absence of ELT. The difference between the two 
groups was statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Suiyeys of MA ELT School Teachers, Spring 2011, Items 5f 9c, 9b, lOe, 10c, lOf, lOh, 10a, lOd, 10b, 
lOg, 9d, 9a; Matched Comparison Teacher Sun’eys, Items 5f 8c, 8b, 9e, 9c, 9f, 9h, 9a, 9d, 9b, 9g, 8d, 8a. 

Sample: 1609 teachers. Nonresponse rates across ELT and matched comparison teacher survey items ranged from 6.7 to 24.6 
percent. 
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Time Available for Planning and Professional Development 

In addition to providing more time for instructional activities, the ELT initiative is intended to provide 
teachers with more time to collaborate with one another and to provide targeted professional development 
to strengthen teachers’ instructional practices. As a result, teachers in ELT schools are expected to be 
more satisfied with the amount of time available for collaborative planning, and professional development 
than would be expected in the absence of ELT 

As hypothesized, teachers in ELT schools were significantly more satisfied with the time available for 
collaborative planning (57 percent) than would be expected without ELT (43 percent; see Exhibit 5.6). 
Further, 70 percent of teachers in ELT schools reported that the length of the school day allows for 
coordination of instruction with other teachers compared to 53 percent in the absence of ELT. 

Conversely, significantly fewer teachers in ELT schools reported that the amount of collaborative 
planning time is a problem area (46 percent) compared to the estimated counterfactual (66 percent). There 
were no statistically significant effects of ELT on teachers’ reported satisfaction with the amount of time 
available for professional development, about 70 percent of the teachers at ELT and MC schools were 
generally satisfied with the time and opportunities available for professional development. 


Exhibit 5.6: Teacher Perceptions of Time Available for Planning and Professional Development 


Percent of 



Respondents 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


Time Available for Planning 


This year, this topic is a problem area: Amount of 
collaborative planning time 

46% 

66% 

-19.6 ** 

0.004 

Length of day allows coordination of instruction 

70 

53 

16.7 ** 

0.008 

Time available for collaborative planning 

57 

43 

14.1 * 

0.038 

Time Available for Professional Development 



Time available for professional development 

67 

72 

-4.8 

0.538 

This year, this topic is a problem area: Professional 
development opportunities 

41 

38 

3.4 

0.556 


EXHIBIT READS: Forty-six percent of teachers in ELT schools agreed that the amount of time for collaborative planning 
was a problem area this year, compared to an estimated 66 percent in the absence of ELT. The difference between the two 
groups was statistically significant. 


Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Items 6i, 5g, 32b, 32c, 6j; Matched Comparison 
Teacher Surveys, Items 6i, 5g, 31b, 31c, 6j. 

Sample: 1609 teachers. Nonresponse rates across ELT and matched comparison teacher survey items ranged from 7.0 to 13.4 
percent. 
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Job Satisfaction 

Given ELT’s expectations for an improved professional culture of leadership and collaboration, along 
with an emphasis on increased time for instructional activities, planning, and professional development, it 
is possible that ELT teachers would report greater job satisfaction than would be expected in the absence 
of ELT. However, results presented in Exhibit 5.7 indicate that there are no statistically significant effects 
of ELT on respondents’ satisfaction with teaching as a profession or with their current teaching position. 
Overall, 95 percent of teachers at all schools agreed or strongly agreed that they were very satisfied with 
being a teacher. Likewise, most teachers reported that they were satisfied with teaching at their current 
school. 


Exhibit 5.7: Teacher Job Satisfaction 


Percent of 



Respondents 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


Satisfaction With Teaching 


If 1 could start over again, 1 would still become a teacher 
Overall, 1 am very satisfied with being a teacher 
Stress and challenges are not worth it 

88% 

95 

15 

89% 

96 

15 

-1.1 

-0.4 

0.2 

0.761 

0.764 

0.941 

Satisfaction With Teaching at This School 





Satisfied with salary 

61 

54 

7.1 

0.261 

Satisfied with teaching at this school 

87 

85 

1.5 

0.725 

Think about transferring 

34 

34 

-0.4 

0.935 


EXHIBIT READS: Eighty-eight percent of teachers in ELT schools agreed that if they could start over again they would still 
become a teacher, compared to an estimated 89 percent in the absence of ELT. The difference between the two groups was not 
statistically significant. 


Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Items Sc, 5a, 5h, 5j, 5b, 5d; Matched Comparison 
Teacher Surveys, Items 5c, 5a, 5h, 5j, 5b, 5d. 

Sample: 1609 teachers. Nonresponse rates across ELT and matched comparison teacher survey items ranged from 5.5 to 1 1.9 
percent. 


Student Non-Academic Outcomes 

Key Findings 

• A significantly higher proportion of teachers in ELT schools reported that teachers and students 
spend sufficient instructional time together than would be expected in the absence of ELT. 

• Significantly more teachers in ELT schools reported that teacher and staff fatigue, as well as 
student fatigue, were problem areas than would be expected without ELT. Likewise, a 
significantly higher proportion of students in ELT schools reported that they were tired in school 
than the counterfactual estimate. 
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• Significantly fewer students in ELT schools reported that: they look forward to going to school; 
like being in school; that all of their classes are important to them; and that they like the length of 
their school day, than would be expected without ELT. 

• A significantly smaller proportion of teachers in ELT schools reported that student academic 
performance and homework completion rates were problem areas than the estimated 
counterfactual. 

• Students in ELT schools had statistically significantly higher suspension rates than would be 
expected in the absence of ELT ; however, while statistically significant, the differences were 
extremely small in magnitude, and therefore are unlikely to have educational or practical 
significance. 

Student-Educator Relationships 

With more time available during the school day, students in ELT schools are hypothesized to spend more 
time interacting with educators and peers. As a result, they may have more opportunities for higher 
quality peer-to-peer and educator-student relationships. To test this hypothesis, teacher and student 
surveys asked respondents about their interactions with one another. Teacher responses are assessed using 
the teacher multi-level models discussed in more detail in the previous section, while student responses 
are assessed using the multi-level models discussed in the section about non-instructional time. Again, 
actual ELT averages are reported relative to the regression-adjusted non-ELT average for selected 
outcomes. 

Results in Exhibit 5.8 suggest that peer-to-peer relationships in ELT schools are not significantly different 
than would be expected in the absence of the initiative. Notably, fewer than half of all students across all 
schools reported that students at their school treat each other well and with respect. 

Results of items regarding educator-student relationships also revealed no significant effects, with one 
exception: 92 percent of teachers in ELT schools reported that teachers and students spend sufficient 
instructional time together compared to 64 percent for the estimated counterfactual. Otherwise, students 
across all schools generally agreed that teachers care about their students, take the time to explain what 
they are teaching and answer their questions in class. Three-quarters of all students also reported that if 
there is something they don’t understand in a class, there is someone in the school who can explain it to 
them. 
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Exhibit 5.8: Student and Teacher Relationships 



Percent of 
Respondents who 
Agree 




Actual 
ELT Mean 

Estimated 

Non-ELT 

Mean 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Student-Teacher Relationships 



1 get along well with my teachers 

59% 

65% 

-5.9 

0.283 

Students treat teachers with respect 1 

49 

44 

5.8 

0.472 

My teachers care about their students 

69 

73 

-3.7 

0.519 

1 feel like 1 can talk to a teacher about my problems 

39 

41 

-1.7 

0.693 

This year, this topic is a problem area: Relationship with 
students 1 

9 

10 

-0.4 

0.862 

Teacher-Student Relationships 



Teachers and students spend sufficient instructional time 
together 1 

92 

68 

24.1 *** 

<.0001 

My teachers take time to explain what they are teaching 
so 1 understand it 

72 

76 

-4.2 

0.297 

1 get my questions answered in this class 

80 

83 

-2.5 

0.318 

Adult-Student Relationships 



There are other people besides my regular teachers in the 
school who have time to give extra help 

48 

51 

-2.8 

0.390 

If 1 do not understand something in this class, there is 
someone in this school 1 can ask to explain it later 

75 

76 

-0.9 

0.718 

Peer Relationships 



Students at my school treat each other well 

43 

41 

2.4 

0.593 

Students treat each other with respect 1 

40 

38 

1.9 

0.793 


EXHIBIT READS: Fifty-nine percent of students in ELT schools indicated that they get along well with their teachers, 
compared to an estimated 65 percent in the absence of ELT. The difference between the two groups was not statistically 
significant. 

Notes: 1 Denotes an item from the teacher survey. All other items are from the student survey. 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Items 7z, 6k, 7a, 7aa; Matched Comparison 
Teacher Surveys, Items 7s, 6k, 7a, 7t. Abt Associates’ Surveys ofMA ELT and Matched Comparison School Students, Spring 
2011, Items 11c, lie, lli, 11a, 9m, 111, 9t, 8m. 

Sample: 1609 teachers, 4840 students. Nonresponse rates across ELT and matched comparison teacher survey items ranged 
from 4.6 to 6.0 percent. Nonresponse rates across ELT and matched comparison student survey items ranged from 2.4 to 3.7 
percent. 


Student Satisfaction and Engagement 

A key expectation for the implementation of ELT is increased student satisfaction and engagement with 
their school. Additional opportunities for enrichment, instruction, and interactions may translate into more 
positive perceptions of school, teachers, and instruction. Likewise, these opportunities could be integral to 
improved academic and behavioral engagement. The following section examines key measures of student 
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satisfaction and engagement, while also addressing several factors that could be barriers to achieving the 
most positive outcomes in these domains. 

Possible Barriers to Student Satisfaction and Engagement 

While ELT is hypothesized to be related to improved student satisfaction and engagement, it is important 
to acknowledge obstacles ELT students and teachers might face in achieving these goals. In particular, 
Exhibit 5.9 highlights two potential physical consequences of a longer school day: fatigue and hunger. 
Indeed, a higher proportion of students and teachers in ELT schools reported that fatigue was a problem 
area compared to the estimated counterfactual. Fifty-three percent of students in ELT schools reported 
that they were tired during class compared to 42 percent in the counterfactual. While fatigue was more 
likely to be reported as a problem amongst teachers and students in ELT schools, there was no statistical 
difference between the percentages of students who reported that they were hungry during class. 
However, nearly half of all students across schools reported that they were hungry during class. 

Notably, findings about teacher and student fatigue closely parallel those from the previous year’s report. 
Actual means, estimated non-ELT means, and estimated differences for teacher reports of teacher/staff 
fatigue and student fatigue are within 5 percentage points of last year’s results. Likewise, the estimated 
difference in student reports of fatigue is similar in magnitude to last year’s finding. The actual ELT and 
estimated non-ELT mean, however, is higher this year (53 and 42 percent, respectively) compared to last 
year (28 and 21 percent, respectively). Because the timing of student survey administration is left to 
teachers’ discretion, it is possible that more surveys were administered at the end of the school day this 
year than last year. 


Exhibit 5.9: Potential Physical Consequences of an Expanded Day 



Percent of 
Respondents 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Actual 
ELT Mean 

Estimated 

Non-ELT 

Mean 

Student Hunger 



1 am hungry in this class 

50% 

45% 

4.7 

0.399 

Teacher/Student Fatigue 



This year, this topic is a problem area: Teacher/staff 
fatigue 1 

This year, this topic is a problem area: Student fatigue 1 
1 am tired during this class 

72 

62 

53 

54 

48 

42 

17.7 ** 

14.5 ** 
11.2 * 

0.006 

0.009 

0.016 

EXHIBIT READS: Fifty percent of students in ELT schools indicated that they were hungry in class, compared to an 


estimated 45 percent in the absence of ELT. The difference between the two groups was not statistically significant. 

Notes: 1 Denotes an item from the teacher survey. All other items are from the student survey. 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Items 6g, 6f; Matched Comparison Teacher 
Surveys, Items 6g, 6f. Abt Associates’ Surveys ofMA ELT and Matched Comparison School Students, Spring 2011, Items 9g, 9f. 
Sample: 1609 teachers, 4840 students. Nonresponse rates across ELT and matched comparison teachers survey items ranged 
from 8.3 to 8.8 percent. Nonresponse rates across ELT and matched comparison student survey items ranged from 1.9 to 2.0 
percent. 
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The significant results highlighted in Exhibit 5.9 are explored further in Exhibit 5.10 by examining 
variation in significant outcomes by school. Each bar in the charts represents average responses for a 
particular ELT school, while the horizontal line represents the overall ELT average for the survey item. 
Note that school numbers presented in this chapter do not correspond with school numbers used in 
Chapter 6. Further, while schools 1 through 15 correspond to the same schools across teacher and student 
surveys, because of slight differences in the teacher and student survey samples 63 , school 16 is not the 
same across survey types and school 17 is only represented in the student survey sample. 

Examining average responses by school highlights school-by-school variation in these items. For 
example, while 72 percent of ELT teachers overall indicated that teacher and staff fatigue was a problem 
in their school, responses ranged widely, from 27 to 93 percent. The range for teacher reports of student 
fatigue is almost as large, spanning from 24 to 79 percent. Finally, although about half of students in ELT 
schools reported being tired in class, these responses ranged from 35 percent to 69 percent, depending on 
the school. 

In addition, these descriptive results also suggest that there is some consistency across responses from 
students and teachers in the same school. For example, teacher and student responses to questions about 
fatigue are consistently lower than average in schools 9, 1 1, and, in particular, 15. Conversely, responses 
are consistently higher than average in schools 7 and 8. 


63 As noted in Chapter 2, because one ELT school does not serve 5 th or 8 th grade students, it was not surveyed and 
only 17 of the 18 ELT schools are represented in this chapter. Further, because two matched comparison school 
teacher survey response rates were below 70 percent, these schools and their ELT pairs were removed from the 
teacher survey analysis sample, resulting in a teacher survey sample of 16 of the 18 ELT schools. 
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Exhibit 5.10: Significant Items Representing Some Potential Physical Consequences of an Expanded Day, by School 


Teacher/staff 
fatigue is a problem area 1 



School 


Student 

fatigue is a problem area * 1 



7 8 9 10 11 12 13 14 15 16 

School 


I am tired during this class 



School 


□ ELT School Average B Overall ELT Average 

EXHIBIT READS: In spring 201 1, 80 percent of teachers in School 1 reported that teacher/staff fatigue was a problem area in their school. Across all 16 ELT schools with 
teacher responses, an average of 72 percent of teachers reported that teacher/staff fatigue was a problem area in their school. 

Note: Schools 1 through 15 correspond to the same schools across teacher and student surveys (e.g., school 1 is the same across all graphs). However, because of slight 
differences in the teacher and student survey samples, school 16 is not the same across survey types and school 17 is only represented in the student survey sample. 

1 Denotes an item from the teacher survey. All other items are from the student survey. 

Source: Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Items 6g, 6f; Abt Associates' Surveys ofMA ELT and Matched Comparison School Students, 
Spring 2011, Item 9f. 

Sample: 918 teachers from ELT schools. 2,300 students from ELT schools. Nonresponse rates across ELT teacher survey items ranged from 6 to 6.5 percent. Nonresponse rate 
for the ELT student survey item was 1.6 percent. 


Abt Associates Inc. 
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Student Satisfaction 

Students reported that they were generally satisfied with their teachers and the instruction at their schools. 
Specifically, over 80 percent reported that they learned a lot in this class; that their teacher helps explain 
difficult concepts; and that their teacher is knowledgeable about the subject matter of the class. As Exhibit 
5.11 indicates, however, significantly fewer ELT students reported that they looked forward to going to 
school, liked being at their school, and that they liked the length of the school day (54 percent versus 66 
percent). 

To explore these results further, Exhibit 5.12 presents average student responses to the significant items 
by school. Similar to Exhibit 5.10, each bar represents average responses for an ELT school, while the 
horizontal line represents the overall ELT average for a particular item. As all items draw from the same 
survey, school numbers across charts are identical and can be compared to assess consistency in 
responses. 

The charts highlight a large amount of variation in responses to items about student satisfaction with 
school and the length of the school day. For example, about half of all ELT students reported that they 
looked forward to going to school and liked being in their school. However, responses by school ranged 
by about 40 percentage points, from 31 to 76 percent for the item asking students if they looked forward 
to going to school and from 39 to 88 percent for the item asking students about how much they liked 
being in their school. The range in responses for the item asking students about how much they liked the 
length of their school day was also about 40 percentage points, with overall responses equaling 22 percent 
and individual school averages spanning from 7 to 46 percent. Finally, note that student responses in 
schools 5, 6, 12, 14 and 15 were consistently above average while responses in schools 7, 8, 13, and 17 
were consistently below average on all of the presented items. 
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Exhibit 5.11: Student Satisfaction and Engagement 


Percent of 



Respondents 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


General Satisfaction With School 


1 look forward to going to school 

54% 

66% 

-12.3 *** 

<.0001 

1 like being at my school 

56 

67 

-11.1 ** 

0.037 

1 am bored in school 

55 

47 

8.4 

0.090 

Satisfaction With Teachers/Instruction 



1 learn a lot in this class 

83 

86 

-2.4 

0.418 

The teacher of this class helps explain difficult concepts 
well 

82 

84 

-2.1 

0.545 

My teacher knows a lot about the subject matter of this 
class 

93 

93 

0.2 

0.921 

Satisfaction With Length of School Day 



1 like the length of my school day this year 

22 

38 

-16.5 ** 

0.006 


EXHIBIT READS: Fifty-four percent of students in ELT schools indicated that they looked forward to going to school, 
compared to an estimated 66 percent in the absence of ELT. The difference between the two groups was statistically significant. 


Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT and Matched Comparison School Students, Spring 2011, Items 8d, 8b, 8c, 9q, 9n, 
9p, 5. 

Sample: 4840 students. Nonresponse rates across ELT and matched comparison student survey items ranged from 1.3 to 2.5 
percent. 
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Exhibit 5.12: Significant Student Satisfaction and Engagement Items, by School 


I look forward to going to school I like being at my school 




School School 


Do you like the length of your school day this year 



School 


□ ELT School Average | Overall ELT Average 

EXHIBIT READS: In spring 201 1, 62 percent of students in School 1 reported that they looked forward to going to school. Across all 17 ELT schools with student responses, an 
average of 54 percent of students reported that they looked forward to going to school. 

Note: School numbers are identical across charts. 

Source: Abt Associates' Surveys ofMA ELT and Matched Comparison School Students, Spring 2011, Items 8d, 8b, 5. 

Sample: 2,300 students from ELT schools. Nonresponse rates across items ranged from .8 to 1.6 percent. 
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Academic Engagement 

Exhibit 5.13 suggests that teacher perceptions of student engagement at ELT schools were significantly 
more positive than would be expected in the absence of ELT. Compared to the counterfactual, 
significantly fewer ELT teachers reported that student academic performance (68 percent versus 83 
percent) and homework completion rates (70 percent versus 82 percent) were problems at their schools. 
However, significantly fewer students at ELT schools reported that all of their classes were important to 
them than would be expected without ELT. To further explore this unexpected finding, Exhibit 5.14 
presents school level ELT averages of the item asking students if all classes are important to them in 
conjunction with the overall ELT average, represented by the horizontal line. Results present the range in 
overall school level responses to the item and indicate that although 65 percent of ELT students reported 
that all of their classes are important to them, this spanned from a low of 34 percent to a high of 90 
percent, depending on the school. 


Exhibit 5.13: Student Academic Engagement 



Percent of 
Respondents 




Actual 
ELT Mean 

Estimated 

Non-ELT 

Mean 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Academic Engagement 



This year, this topic is a problem area: Student academic 
performance 1 

68% 

83% 

-15.1 * 

0.011 

This year, this topic is a problem area: Homework 
completion rates 1 

70 

82 

-12.1 * 

0.028 

All of my classes are important to me 

65 

73 

-7.9 * 

0.032 

This year, this topic is a problem area: Student 
engagement 1 

67 

74 

-7.1 

0.210 

1 pay attention in this class 

85 

89 

-4.1 

0.185 

1 am interested in the work 1 get to do in this class 

68 

71 

-3.0 

0.351 


EXHIBIT READS: Sixty-eight percent of teachers in ELT schools agreed that student academic performance was a problem 
area, compared to an estimated 83 percent in the absence of ELT. The difference between the two groups was statistically 
significant. 

Notes: 1 Denotes an item from the teacher survey. All other items are from the student survey. 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Items 6a, 6c, 6b: Matched Comparison Teacher 
Sun’eys, Items 6a, 6c, 6b. Abt Associates' Sun’eys ofMA ELT and Matched Comparison School Students, Spring 2011, Items 
8i, 9c, 9a. 

Sample: 1609 teachers, 4840 students. Nonresponse rates across ELT and matched comparison teachers survey items ranged 
from 7.7 to 19.0 percent. Nonresponse rates across ELT and matched comparison student survey items ranged from 1.6 to 2.0 
percent. 
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Exhibit 5.14: Percent of Students Reporting that “All of my classes are important to me,” by School 


All of my classes are important to me 



School 

Q ELT School Average | Overall ELT Average 

EXHIBIT READS: In spring 2011, 70 percent of students in School 1 agreed with the statement "All of my classes are important to me.” Across all 17 ELT schools with 
student responses, an average of 65 percent of students agreed with the statement “All of my classes are important to me.” 

Note: School numbers are identical across charts. 

Source: Abt Associates’ Surveys of MA ELT and Matched Comparison School Students, Spring 2011, Item 8i. 

Sample: 2,300 students from ELT schools. Nonresponse rate was 1.4 percent. 
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Behavioral Engagement 

Results from the student and teacher surveys presented in Exhibit 5.15 suggest that the majority of 
students across all schools had high levels of particular types of behavioral engagement. Almost 90 
percent of students reported that they followed class rules and only one-quarter reported getting in trouble 
in class. Teachers across all schools, however, indicated that they felt that student behavior was a problem 
area in their schools. 


Exhibit 5.15: Behavioral Engagement from Survey Items 


Percent of 



Respondents 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


Behavioral Engagement 


This year, this topic is a problem area: Student behavior 1 

76% 

84% 

-7.4 

0.336 

1 follow the rules in this class 

87 

89 

-2.0 

0.452 

1 get in trouble in this class 

26 

25 

0.9 

0.706 


EXHIBIT READS: Seventy-six percent of teachers in ELT schools agreed that student behavior was a problem area, 
compared to an estimated 84 percent in the absence of ELT. The difference between the two groups was not statistically 
significant. 

Notes: 1 Denotes an item from the teacher survey. All other items are from the student survey. 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Item 6d ; Matched Comparison Teacher Surveys, 
Item 6d. Abt Associates’ Surveys ofMA ELT and Matched Comparison School Students, Spring 2011, Items 9k, 9b. 

Sample: 1609 teachers, 4840 students. The nonresponse rate across ELT and matched comparison teacher survey item 6d 
equaled 5.5 percent. Nonresponse rates across ELT and matched comparison student survey items ranged from 1.8 to 2.3 
percent. 


Behavioral engagement was also examined using SIMS data collected by ESE on attendance rates, in-and 
out-of-school suspensions, and truancy rates. Recall that while survey results are based on 2010-1 1 cross- 
sectional data, SIMS data are longitudinal, allowing for analysis of behavioral outcomes after one, two, 
three, and four years of ELT implementation. Because SIMS data are collected from the same students 
across multiple school years, student fixed effects as well as school and year fixed effects are included in 
the behavioral outcome models. These models also include student-level covariates and appropriately 
account for the fact that students are clustered within schools. 

The estimated effects of ELT on the SIMS behavioral indicators are presented in Exhibit 5.16. The 
estimated impacts suggest that there were no effects of ELT on attendance rates, as estimated impacts in 
three of the four years were not statistically significant. After one year of implementation, the average 
student attendance rate for students in ELT schools was 93.70 percent, compared to 94 percent in the 
absence of ELT. Although the estimated difference of 0.27 percentage points was statistically significant, 
its magnitude is unlikely to be practically meaningful as a difference of this magnitude represents about 
three hours of attendance per student over an entire school year. 64 Considering that students attending 
ELT schools spent an additional 90 minutes in school each day, on average, or an additional 266 hours 


64 This assumes a 180-day school year and a 6-hour school day, an approximate average in non-ELT schools. 
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(the rough equivalent of over 40 non-ELT school days) per year, this difference is not practically 
significant. The smaller estimated differences in attendance rates in the second through fourth years of 
implementation were not statistically significant. 


Exhibit 5.16: Behavioral Engagement from SIMS Data 


Percent of Students 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


Year 1 


Attendance rate 

93.70% 

93.97% 

-0.27 *** 

<0.001 

In-school suspension rate 

0.03 

0.04 

-0.01 *** 

<0.001 

Out-of-school suspension rate 

0.10 

0.08 

0.02 *** 

<0.001 

Truancy rate 

0.57 

0.62 

-0.05 * 

0.020 

Year 2 



Attendance rate 

93.90 

93.96 

-0.06 

0.326 

In-school suspension rate 

0.06 

0.03 

0.03 *** 

<0.001 

Out-of-school suspension rate 

0.10 

0.08 

0.02 *** 

<0.001 

Truancy rate 

0.61 

0.65 

-0.04 

0.095 

Year 3 



Attendance rate 

93.99 

94.02 

-0.03 

0.677 

In-school suspension rate 

0.08 

0.01 

0.07 *** 

<0.001 

Out-of-school suspension rate 

0.13 

0.07 

0.06 *** 

<0.001 

Truancy rate 

0.79 

0.74 

0.05 

0.075 

Year 4 



Attendance rate 

94.14 

94.13 

0.01 

0.925 

In-school suspension rate 

0.04 

-0.01 

0.05 *** 

<0.001 

Out-of-school suspension rate 

0.10 

0.05 

0.05 *** 

<0.001 

Truancy rate 

0.77 

0.79 

-0.02 

0.485 


EXHIBIT READS: At the end of the first year of ELT implementation, the average attendance rate at ELT schools was 93.70 
percent, compared to an estimated 94 percent in the absence of ELT. The difference between the two groups was statistically 
significant. 


Notes: 

* statistically significant at p < .05 level 

** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 
Source: Individual student records obtained from MA ESE. 


Sample: Student records from ELT and matched comparison schools, cohorts 1-3 (n=49) from 2001-02 to 2010-11 for 
attendance and 2003-04 to 2010-11 for other measures: 257,348 records for attendance and 205,990 for other measures. Fewer 
than 1 percent of students were eliminated from the analytic sample due to missing covariates. 


The estimated percentages of students receiving in- and out-of-school suspensions during the first four 
years of implementation, while statistically significant, were extremely small and unlikely to have 
educational or practical significance. In the first year of ELT implementation, significantly fewer students 
in ELT schools received in-school suspensions than would have been expected in the absence of ELT. In 
the second, third, and fourth years of implementation, the difference reversed, such that students in ELT 
schools had significantly more in-school suspensions compared to the counterfactual. Students in ELT 
schools also had more out-of-school suspensions than would be expected in the absence of ELT, with 
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differences ranging from .02 percentage points in the first year of implementation to .06 percentage points 
in the third year of implementation. 

Finally, results suggest that there were no effects of ELT on truancy rates, as estimated impacts in three of 
the four implementation years were not statistically significant. In the first year of implementation, ELT 
schools had a significantly lower truancy rate than the estimated rates in the absence of ELT. 


Summary 

The analyses presented in this chapter suggest that the ELT school day has some important implications 
for teacher and student outcomes. ELT schools spend significantly more time on most core academic 
subjects and enrichment activities than do matched comparison schools. Presumably, these increases in 
instructional time are related to higher ELT teacher satisfaction for instructional time, collaborative 
planning time, the ability to spend sufficient instructional time with students, and student academic 
engagement, compared to what would be expected in the absence of ELT. 

Flowever, the results also indicate that the increased school day may heighten fatigue, as significantly 
more teachers and students in ELT schools reported being tired than would be expected in the absence of 
ELT. Fatigue may be related to student reports in ELT schools of significantly lower levels of satisfaction 
in several areas, including liking the length of their school day or looking forward to going to school. 
Descriptive analyses, however, highlight the variation in ELT school-level responses to these items, and 
suggest that ELT teachers and students vary widely by school in their perceptions. 

Finally, while students in ELT schools had statistically significantly higher suspension rates than would 
be expected in the absence of ELT, the differences were extremely small in magnitude, and therefore are 
unlikely to have educational or practical significance. 
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Chapter 6. Effect of ELT on School Characteristics and Student 
Achievement 


This chapter examines the impact of ELT implementation on school characteristics and student 
achievement. It begins by assessing the relationship between ELT and the characteristics of study schools, 
and exploring differences between ELT and matched comparison schools in student mobility. The focus 
of the chapter then shifts to student achievement results, beginning with a presentation of descriptive 
MCAS data and then turning to more rigorous impact estimation that relies upon a strong quasi- 
experimental design. The chapter also presents results of exploratory analyses that examine variation in 
student achievement across the schools participating in the initiative as well as the relationship between 
levels of implementation and student achievement. The findings draw from state-provided school-level 
data to describe school characteristics, and from state -provided individual student-level data to describe 
student mobility and model the impact of ELT on student achievement patterns. 

Recall that the impact models used in this chapter leverage pre-ELT data to control for observable and 
unobservable, stable characteristics of schools and data from matched comparison schools to control for 
year-to-year variation in outcomes (see Chapter 2 for more detail). These models, therefore, control for 
many of the alternative hypotheses that might explain observed differences and represent strong, quasi- 
experimental analyses. It is important to note that impact models estimate the effect of ELT on selected 
outcomes by essentially comparing the observed outcomes for ELT schools and students and the 
counterfactual outcomes, which are estimated using the comparison group to capture what would have 
occurred in the absence of ELT. 


Key Findings 

• In the first and second years of implementation, ELT schools served a statistically significantly 
greater proportion of minority students than estimated in the absence of ELT, although the 
estimated magnitude of the differences (3.7 and 4.0 percentage points, respectively) is unlikely to 
be practically meaningful, and there were no effects of ELT on schools’ minority student 
population in the third or fourth year of implementation. 

• In the third year of implementation, ELT schools had a statistically significantly smaller 
proportion of highly qualified core academic teachers compared to the estimated proportion in the 
absence of ELT (2.9 percentage points). 

• In the fourth year of implementation, ELT schools had a statistically significantly lower number 
of FTE teachers (4.7 fewer), and statistically significantly higher student-teacher ratio (almost 
two more students per teacher) than estimated in the absence of ELT. 

• Across all years of implementation, there were no significant differences in average student 
mobility rates between ELT and matched comparison schools. 

• Descriptive analyses restricted to ELT schools indicated variation in student performance levels 
among schools both before implementation began and in the most recent school year (2010-11), 
and indicated no consistent patterns of results. 
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• On average, there were no statistically significant effects of ELT after one, two, three, or four 
years of implementation on MCAS student achievement test outcomes for 3 rd , 4 th , or 7 th grade 
ELA, 4 th , 6 th , or 8 th grade math, or 8 th grade science. 

• There was a statistically significant positive effect of ELT after four years of implementation on 
the MCAS 5 th grade science test. 

• Exploratory analyses using data from non-ELT schools in ELT districts, and non-ELT schools 
statewide, rather than the study’s matched comparison schools, were generally consistent with the 
primary analysis, including the significant finding for 5 th grade science noted above. In addition, 
there was a statistically significant negative effect of ELT on 3 ld grade reading after two years of 
implementation, and there were statistically significant positive effects of ELT on 6 th grade math 
and 8 th grade science after four years of implementation in both the district-level and state-level 
analyses. The state-level analysis also found a statistically significant positive effect of ELT on 
7 th grade ELA after one year of implementation and on 6 th grade math after three years of 
implementation. 

• Exploratory descriptive analysis linking the level of implementation in ELT schools and student 
achievement outcomes indicated no clear patterns or meaningful relationships. 

• Exploratory analysis investigating the difference of the effect of ELT in higher- versus lower- 
implementing schools indicates minimal heterogeneity in the effect by the level of ELT 
implementation. However, the effect of ELT on 8 th grade math in higher-implementing schools is 
estimated to be statistically significantly greater than the effect of ELT in low- implementing 
schools after three and four years of implementation. 


Effects of ELT on School Characteristics and Student Mobility 

School Characteristics 

A school’s adoption of ELT could potentially result in parents and students revisiting their choices of 
where children attend school assuming that they have some choice. Similarly, teachers may reconsider 
where to teach, either by choosing to join an ELT school or to leave a school in the early stages of 
implementing ELT. Exhibit 6. 1 explores these possibilities by summarizing school-level averages of 
student and teacher characteristics after one, two, three, and four years of implementation in ELT schools 
compared to what would have been expected in the absence of ELT. 65 The results present the actual 
observed mean for ELT schools, compared to the average outcomes for the non-ELT counterfactual, 
which is calculated by subtracting the estimated ELT effect from the ELT mean, thereby representing 
what is estimated in the absence of, or without, ELT. 


65 The number of schools included in analyses by implementation year includes only those schools that had 
implemented ELT for that number of (implementation) years. For example, if a school implemented ELT for 
two years, and then exited the initiative, that school would be included in the analyses of impacts after one and 
two years of implementation, and would be excluded from impact estimates after three and four years of 
implementation. 
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Exhibit 6.1 : Effect of ELT on Student and Teacher Characteristics, across Cohorts, by 
Implementation Year 



Actual 

Estimated 

Estimated 



ELT Mean 

Non-ELT Mean 

Difference 

p-value 


Implementation Year 1 


Student Population 





Student enrollment 

516 

510 

6.46 

0.879 

Percent low income 

70% 

71% 

-0.98 

0.573 

Percent minority 

61% 

57% 

3.68" 

0.002 

Percent male 

52% 

52% 

-0.71 

0.235 

Percent special education 

18% 

19% 

-0.64 

0.388 

Percent limited English proficient 

14% 

15% 

-0.42 

0.77 

Percent first language not English 
Teacher Population 

37% 

36% 

0.53 

0.706 

Number of FTE Teachers 

40 

38 

1.74 

0.43 

Percent of teachers licensed in their teaching 
assignment 

97% 

98% 

-0.59 

0.609 

Percent of core academic teachers highly 
qualified 

96% 

97% 

-1.32 

0.26 

Student-teacher ratio 

13 

13 

0.24 

0.573 


Implementation Year 2 


Student Population 





Student enrollment 

533 

522 

10.14 

0.815 

Percent low income 

74% 

75% 

-1.01 

0.632 

Percent minority 

64% 

60% 

3.98* 

0.015 

Percent male 

53% 

52% 

0.14 

0.862 

Percent special education 

19% 

19% 

0.04 

0.965 

Percent limited English proficient 

16% 

17% 

-0.46 

0.81 

Percent first language not English 
Teacher Population 

39% 

38% 

0.78 

0.693 

Number of FTE Teachers 

40 

38 

2.24 

0.369 

Percent of teachers licensed in their teaching 
assignment 

98% 

99% 

-1.46 

0.327 

Percent of core academic teachers highly 
qualified 

96% 

98% 

-2.44 

0.088 

Student-teacher ratio 

13 

13 

0.4 

0.439 


Implementation Year 3 


Student Population 





Student enrollment 

543 

546 

-3.04 

0.941 

Percent low income 

78% 

79% 

-1.63 

0.476 

Percent minority 

64% 

61% 

2.93 

0.136 

Percent male 

53% 

53% 

0.17 

0.854 

Percent special education 

18% 

19% 

-0.31 

0.768 

Percent limited English proficient 

17% 

17% 

-0.11 

0.963 

Percent first language not English 
Teacher Population 

38% 

38% 

0.14 

0.955 

Number of FTE Teachers 

41 

40 

1.03 

0.672 

Percent of teachers licensed in their teaching 
assignment 

98% 

101% 

-3.04 

0.08 

Percent of core academic teachers highly 
qualified 

96% 

99% 

-2.94* 

0.05 

Student-teacher ratio 

13 

13 

0.54 

0.271 


pg. 1 20 | Chapter 6. Effect of ELT on School Characteristics and Student Achievement Abt Associates Inc. 




ELT Evaluation Year 5 Integrated Report 


Exhibit 6.1 : Effect of ELT on Student and Teacher Characteristics, across Cohorts, by 
Implementation Year 



Actual 

Estimated 

Estimated 



ELT Mean 

Non-ELT Mean 

Difference 

p-value 


Implementation Year 4 


Student Population 





Student enrollment 

550 

600 

-49.76 

0.219 

Percent low income 

76% 

80% 

-4.09 

0.205 

Percent minority 

64% 

63% 

0.89 

0.727 

Percent male 

52% 

52% 

-0.91 

0.385 

Percent special education 

20% 

20% 

0.08 

0.955 

Percent limited English proficient 

17% 

16% 

1.02 

0.742 

Percent first language not English 
Teacher Population 

35% 

35% 

0.33 

0.92 

Number of FTE Teachers 

40 

44 

-4.71* 

0.017 

Percent of teachers licensed in their teaching 
assignment 

98% 

100% 

-2.67 

0.281 

Percent of core academic teachers highly 
qualified 

96% 

99% 

-3.48 

0.129 

Student-teacher ratio 

14 

12 

1.70** 

0.004 


EXHIBIT READS: After the first year of ELT implementation, ELT schools had an average enrollment of 516 students, 
compared to an estimated 510 students in the absence of ELT. The difference was not statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: School-level data tables downloaded from the MA ESE website (http://profiles.doe.mass.edu/). 

Sample: Full sample of 49 schools, including 24 ELT and 25 matched comparison schools. Student population outcomes are 
based on data from 2001-02 to 2010-11, including 496 records. Teacher population outcomes are based on data from 2003-04 
to 2010-11, including 396 records. 


In the first and second years of implementation, ELT schools served a significantly greater proportion of 
minority students than would be expected in the absence of ELT. However, the magnitude of these 
differences, 3.7 and 4 percentage points, respectively, are unlikely to be practically meaningful, and the 
differences are not significant in the third and fourth years of implementation. There were no effects of 
ELT within any implementation year on the remaining characteristics of schools’ student populations 
examined, including student enrollment, proportion of students served from low income families, 
proportion male, proportion receiving special education services, proportion designated as limited English 
proficient, and proportion whose first language was not English. 

In the first and second years of implementation, there was no effect of ELT on any of the teacher 
population characteristics examined: number of full-time equivalent (FTE) teachers, proportion of 
teachers licensed in their respective teaching assignments, proportion of teachers in core academic 
subjects designated as highly qualified, 66 or student-teacher ratio. However, in the third year of 


66 According to the No Child Left Behind Act (NCLB), highly qualified teachers must: 1) have a bachelor's 
degree, 2) have full state certification or licensure, and 3) prove that they know each subject they teach. 
(Definition downloaded December 28, 2011, from http://www2.ed.gov/teachers/nclbguide/nclb-teachers- 
toolkit.pdf). 
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implementation, ELT schools had a statistically significantly smaller proportion (2.9 percentage points) of 
highly qualified core academic teachers, compared to the estimated proportion in the absence of ELT. 

This difference likely represents less than one highly qualified FTE teacher, on average, in schools with 
an average of 40 FTE teachers, and while that difference may be educationally meaningful, it should be 
noted that the proportion of highly qualified teachers in ELT schools was over 95 percent. There were no 
other significant differences in teacher population characteristics in the third year of implementation. 
Finally, in the fourth year of implementation, ELT schools were estimated to have a statistically 
significantly lower number (4.7) of FTE teachers than would be expected in the absence of ELT. In what 
may be a related finding, the student-teacher ratio was also statistically significantly higher (almost two 
more students per teacher) in ELT schools than estimated in the absence of ELT. 

Student Mobility 

Another way to gauge whether parents and students are revisiting any choice available of where to attend 
school after ELT implementation is to examine student mobility. The adoption of ELT could increase the 
number of students who enter schools in non-entry grades (e.g., the percentage of students entering the 7 th 
grade in a school that serves 6 th -8 th grade students) or the number of students who leave ELT schools in 
non-exit grades (e.g., leaving the school in the 7 th grade when the school spans from the 6 th to 8 th grade). 

In order to better understand these patterns, the study used student-level annual enrollment data to 
calculate each school’s average student mobility rates in non-entry and non-exit grades for each 
implementation year. As described in more detail in Chapter 2, student mobility averages for ELT and 
matched comparison schools are compared using two-tailed independent samples t-tests that assume equal 
variance between samples. 

Exhibit 6.2 presents the results of examining the percentage of students in non-entry grades across cohorts 
in the first, second, third, and fourth year of ELT implementation who had attended the same school in the 
prior year.to learn whether new students were entering ELT schools at differential rates relative to the 
matched comparison schools. 67 The results suggest that, on average across cohorts and implementation 
years, over 75 percent of students in non-entry grades had attended the same school in the prior year in 
both ELT and matched comparison schools. Student stability levels were slightly higher in matched 
comparison schools, though these differences were not statistically significant. The study also examined 
mobility in non-exit grades at ELT schools compared to matched comparison schools in each year of 
implementation (Exhibit 6. 3). 68 Although the proportion of students in non-exit grades who left ELT 
schools is slightly higher relative to matched comparison schools, the difference is not statistically 
significant. 


67 This analysis excludes students whose prior year’s grade level was not in the same school, which is the case for 
all entry year grades (e.g., the lowest grade served in a school) and if a school’s grade configuration changed 
(e.g., from a 4 th -5 th grade school to a 3 rd -5 th grade school). 

6S This analysis excludes students who could not have stayed in the same school the next year (e.g., the highest 
grade in each school, grades for which the higher grade is not present in the following year due to changes in 
school grade configuration). 
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Exhibit 6.2: Percent of Students who Attended the Same School in the Prior Year, by 
Implementation Year 



Actual 
ELT Mean 

Actual 
MC Mean 

Statistical 

Significance 

(p-value) 

Implementation Year 1 

77% 

79% 

0.585 

Implementation Year 2 

75 

77 

0.588 

Implementation Year 3 

78 

79 

0.792 

Implementation Year 4 

79 

81 

0.566 


EXHIBIT READS: After one year of ELT implementation, 77 percent of ELT students in non-entry grades attended the 
same school in the previous year, compared to 79 percent of students in matched comparison schools. The difference 
between the two groups was not statistically significant. 

Notes: Students whose grade level in the previous year was not available at their current school are excluded. P-values are 
from two-tailed independent samples t-tests that compare school-level mobility percentages by ELT status. 

* statistically significant at p < .05 level 

** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE. 

Sample: Number of students by implementation year: 1st year: 21,635 students (10,438 ELT, 1 1,197 matched comparison); 
2nd year: 19,286 students (9,401 ELT, 9,885 matched comparison); 3rd year: 18,572 students (9,000 ELT, 9,572 matched 
comparison); 4th year: 1 1,255 students (5,802 ELT, 5,453 matched comparison). 


Exhibit 6.3: Percent of Students who Left Schools in Non-Exit Grades, by Implementation 
Year 



Actual 
ELT Mean 

Actual 
MC Mean 

Statistical 

Significance 

(p-value) 

Implementation Year 1 

21% 

17% 

0.070 

Implementation Year 2 

21 

19 

0.405 

Implementation Year 3 

21 

20 

0.510 

Implementation Year 4 

20 

21 

0.823 


EXHIBIT READS: After one year of ELT implementation, 21 percent of ELT students in non-exit grades attended a 
different school in the following year, compared to 17 percent of students in matched comparison schools. The difference 
between the two groups was not statistically significant. 

Notes: Students who could not have stayed in the same school the next year are excluded. P-values are from two-tailed 
independent samples t-tests that compare school-level mobility percentages by ELT status. 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE. 

Sample: Number of students in non-exit grades by implementation year: 1st year: 21,276 students (10,303 ELT. 10,973 
matched comparison); 2nd year: 19,160 students (9,439 ELT, 9,721 matched comparison); 3rd year: 12,513 students (6,398 
ELT, 6,1 15 matched comparison); 4th year: 6,51 1 students (3,555 ELT, 2,956 matched comparison). 


In sum, the results of models examining the relationship between the implementation of ELT and school 
characteristics and student mobility suggest that the characteristics of students and teachers after ELT 
implementation have remained quite stable and that student mobility is equivalent to that in matched 
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comparison schools. As such, these analyses provide some evidence that any observed differences found 
among the study’s outcomes, including the MCAS results presented in the following section, are not 
simply due to observed changes in schools’ student and teacher populations. 


Student Performance Levels in ELT Schools 

The remainder of this chapter explores the relationship between ELT and one of the key expected 
outcomes of the ELT initiative: student achievement outcomes. It begins by presenting descriptive data on 
students’ academic achievement to illustrate patterns in performance at two points in time and between 
ELT and matched comparison schools using various MCAS score metrics: proficiency rates and student 
growth percentiles (SGP). These descriptive analyses document important patterns among ELT and 
matched comparison schools. While the descriptive results can provide valuable information about the 
contexts within which ELT and comparison schools operate, any observed differences between ELT and 
matched comparison schools cannot be attributed to the ELT initiative within the study’s causal 
framework (e.g., the interrupted time series design). These descriptive analyses are then followed by 
results from short interrupted time series models that estimate the causal impact of ELT on MCAS scores 
across cohorts after one, two, three and four years of implementation, using a variety of rigorous quasi- 
experimental controls. Finally, the chapter concludes with an exploratory analysis linking levels of ELT 
implementation to MCAS scores. 

Describing ELT School-level Student Achievement 

Descriptive analyses first examine variation in ELT schools’ student achievement. Exhibit 6.4 displays 
the pre-ELT average student performance level for a given grade and subject assessed on the MCAS 
(represented by light blue), and the 2010-11 average performance level for that grade/subject (dark 
purple). Note, however, that the 2010-11 averages correspond to different implementation years, ranging 
from three through five. 

The charts in Exhibit 6.4 present raw percentages of students scoring at the proficient and advanced levels 
on the MCAS tests, and therefore do not reflect the rigorous statistical controls incorporated into the short 
interrupted time series with a comparison group models used in other analyses. Nonetheless, the charts 
provide a snapshot of test scores immediately prior to implementation and in the most recent school year, 
and can help set the stage for the rigorous impact models discussed below. 

Exhibit 6.4 illustrates two important features: 

• There is variation across schools in the percentage of students scoring at the proficient and 
advanced levels both in the pre-implementation and 2010-11 school years. 

• There is variation across schools in the change between the percent of proficient and advanced 
students between the pre-implementation year and the 2010-11 school year. 
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Exhibit 6.4: Panel 1: ELT Schools’ Student Performance Levels, for Last Pre-ELT Year and 2010-11, Elementary Grades 
ELA Grade 3 ELA Grade 4 ELA Grade 5 





Math Grade 3 


Math Grade 4 


Math Grade 5 





Science Grade 3 


No Science Grade 6 
MCAS Test 


Science Grade 4 


No Science Grade 7 
MCAS Test 


Science Grade 5 



P] Last pre-ELT school year P] 2010-2011 school year 

EXHIBIT READS: In the last year prior to ELT implementation, 24 percent of 3rd grade students in School 1 scored proficient or advanced on the ELA portion of the MCAS 
test. In spring 2011, 47 percent of 3rd grade students in School 1 scored proficient or advanced on their ELA test. 

Notes: The first bar in each pair indicates the percentage of students proficient or advanced on the MCAS test for the given outcome in the last year prior to ELT implementation 
for schools; the second bar indicates the percent proficient or advanced in the 2010-11 school year. Schools missing data in 201 1 or in the year prior to implementation are 
excluded from the charts. School numbers identify the same school across charts. 

Source: Individual student records obtained from MA ESE. 


Abt Associates Inc. 
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Exhibit 6.4: Panel 2: ELT Schools’ Student Performance Levels, for Last Pre-ELT Year and 2010-11, Middle School Grades 
ELA Grade 6 ELA Grade 7 ELA Grade 8 



4 5 6 7 8 9 10 11 12 13 14 15 16 




Math Grade 6 


Math Grade 7 


Math Grade 8 



4 5 6 7 8 9 10 11 12 13 14 15 16 


Science Grade 6 


No Science Grade 6 
MCAS Test 



Science Grade 7 


No Science Grade 7 
MCAS Test 



Science Grade 8 



Q Last pre-ELT school year | 2010-2011 school year 

EXHIBIT READS: In the last year prior to ELT implementation, 45 percent of 6th grade students in School 4 scored proficient or advanced on the ELA portion of the MCAS 
test. In spring 201 1, 35 percent of 6th grade students in School 4 scored proficient or advanced on their ELA test. 

Note: The first bar in each pair indicates the percentage of students proficient or advanced on the MCAS test for the given outcome in the last year prior to ELT implementation 
for schools; the second bar indicates the percent proficient or advanced in the 2010-1 1 school year. Schools missing data in 201 1 or in the year prior to implementation are 
excluded from the charts. School numbers identify the same school across charts. 

Source: Individual student records obtained from MA ESE. 
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The percentage of proficient and advanced students varies considerably across schools, illustrating no 
clear pattern in performance levels at either time point represented. Furthermore, there is variation across 
schools in the change between the two time points; some schools have substantially increased the 
percentage of their students reaching the proficient or advanced performance levels, some schools have 
seen little change in the percentage of their students reaching the proficient or advanced performance 
levels, and in some schools the percentage of students reaching the proficient or advanced performance 
levels has decreased when comparing the pre-implementation year and 2010-11 school year. 

To present a different perspective on these results, Exhibit 6.5 displays the proficiency data by school, 
rather than by MCAS test. Panel 1 includes information on the elementary schools, Panel 2 on K-8 
schools, and Panel 3 on middle schools. These charts suggest that there was an increase in the percentage 
of proficient or advanced students between time points on the majority of MCAS tests. 
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Exhibit 6.5: Panel 1: Individual ELT Schools, Percent of Students Scoring Proficient or Advanced on MCAS Tests by Subject and 
Grade, Last Year Prior to ELT Implementation and 2010-11, Elementary Schools 
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EXHIBIT READS: In the last year prior to ELT implementation, 24 percent of 3rd grade students in School 1 scored proficient or advanced on the EL A portion of the MCAS 
test. In spring 2011, 47 percent of 3rd grade students in School 1 scored proficient or advanced on their ELA test. 

Note: The first column in each set represents the last year prior to ELT implementation, and the second column the 2010-11 (most recently completed) school year. 

Source: Individual student records obtained from MA ESE. 
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Exhibit 6.5: Panel 2: Individual ELT Schools, Percent of Students Scoring Proficient or Advanced on MCAS Tests by Subject and 
Grade, Last Year Prior to ELT Implementation and 2010-11, K-8 Schools 

School 6 School 7 




School 8 School 9 




Q Last pre-ELT school year B 2010-2011 school year 

EXHIBIT READS: In the last year prior to ELT implementation, 33 percent of 3rd grade students in School 6 scored proficient or advanced on the ELA portion of the 
MCAS test. In spring 201 1, 67 percent of 3rd grade students in School 6 scored proficient or advanced on their ELA test. 

Note: The first column in each set represents the last year prior to ELT implementation, and the second column the 2010-11 (most recently completed) school year. 
Source: Individual student records obtained from MA ESE 
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Exhibit 6.5: Panel 3: Individual ELT Schools, Percent of Students Scoring Proficient or Advanced on MCAS tests by Subject and 
Grade, Last Year Prior to ELT Implementation and 2010-11, Middle Schools 

School 10 School 11 School 12 



School 13 



School 14 



School 15 



School 16 
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Q Last pre-ELT school year 


j 2010-2011 school year 


EXHIBIT READS: In the last year prior to ELT implementation, 18 percent of 5th grade students in School 10 scored proficient or advanced on the ELA portion of the MCAS 
test. In spring 2011, 36 percent of 5th grade students in School 10 scored proficient or advanced on their ELA test. 

Note: The first column in each set represents the last year prior to ELT implementation, and the second column the 2010-11 (most recently completed) school year. 

Source: Individual student records obtained from MA ESE. 
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Alternative MCAS Metrics: Student Growth Percentiles for ELT and Matched Comparison Schools 

Starting in 2008, Massachusetts began assessing student achievement outcomes using student growth 
percentiles (SGPs). This alternative achievement metric represents school-level median growth in 
students’ performance from one year to the next on those MCAS tests administered in consecutive school 
years. Given this metric’s potential for providing additional important context for the study, this section 
presents median student growth percentiles comparing achievement in ELT and matched comparison 
schools. 

Unlike performance levels, which indicate how a student scored relative to state grade-level standards in 
the year of the test, student growth percentiles quantify how much each student improved over her/his 
past performance, relative to other students with a similar test score history (known as the student’s 
“academic peers”). In this way, regardless of any individual student’s performance level on the MCAS in 
a given year, ajf students have the opportunity to display growth anywhere from the 1 sl to 99 th percentile 
in a subsequent year. For example, an individual student with an SGP of 55 represents improved 
performance that is more than 55 and less than 45 percent of that student’s academic peers, regardless of 
that student’s actual performance level. 

When considering comparisons between groups, such as ELT and matched comparison schools, or 
between individual schools, the basis for comparison for each group is the median SGP, that is, the level 
of SGP at which half of the individual students in the given group have a higher, and half have a lower 
SGP. A median SGP for any particular group is always interpreted the same way: if the group has a 
median SGP above 50, then that group of students, on average, improved in performance more than their 
academic peers. A key point in such comparisons is that the reference group is always students’ academic 
peers. “Typical” growth, for individual students and groups, is between the 40 th and 60 th percentiles, with 
values outside that range are considered higher or lower than typical growth. Differences (between 
individual students or median SGPs for groups) of fewer than 10 SGP points are considered unlikely to be 
educationally meaningful. 

Student growth percentiles are a useful tool for understanding student academic progress, yet one must be 
cautious about attributing apparent differences in SGPs between ELT and matched comparison schools to 
ELT. First, SGPs are only available in the post-ELT period (i.e., 2008 to 201 1) and, therefore, may not 
reflect inherent differences between the ELT and matched comparison schools. Second, while SGPs 
represent changes from the previous year, they may not reflect change from the pre-ELT to post-ELT 
periods or during the post-ELT period. Finally, the reference groups used to create the percentiles are not 
explicitly known to the evaluation team, and are likely not similar to the study comparison group for 
which a rigorous matching process determined adequate comparisons for ELT schools. 

Exhibit 6.6 presents median SGPs for ELT and matched comparison schools for ELA (Panel 1) and Math 
(Panel 2) for grades 4 through 8. On average, ELT schools had higher median SGPs than their matched 
comparison schools on all 2011 MCAS tests in each grade tested. 
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Exhibit 6.6: ELT and Matched Comparison Median Student Growth by Grade, 2010-11 

Panel 1: ELA 



Panel 2: Math 



EXHIBIT READS: In spring 2011, the median student growth percentile on the 4 lh grade ELA portion of the MCAS test was 
56 across all ELT schools, and 48 across all matched comparison schools. 

Source: Individual student records obtained from MA ESE. 

Sample: Current ELT schools as of the 2010-11 school year and their matched comparison schools. 


Some differences between median SGPs for ELT and matched comparison schools appeared large, and 
therefore the study conducted exploratory analyses to investigate whether these differences were 
statistically significant. These cross-sectional analyses used data from the 2010-11 school year, control 
for the matched comparison design of the study, and account for clustering of students within schools. 
Exhibit 6.7 presents the mean differences between ELT and matched comparison schools in terms of 201 1 
student growth percentiles (SGP); two of the ten outcomes tested are statistically different: grade 8 ELA 
and grade 4 math. 
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The results from these exploratory analyses should be interpreted with caution, as they are based on cross- 
sectional date (201 1 MCAS scores only), and cannot therefore benefit from the strength of models that 
leverage pre-ELT data to control for observable and unobservable, stable characteristics of schools, and 
data from matched comparison schools to control for year to year variation in outcomes. It is possible that 
over time, one could use SGPs in more robust models, though for this study, there are no SGP data 
available for pre-ELT years. The results are suggestive of potentially meaningful differences in terms of 
student growth between ELT and matched comparison schools in 201 1. 


Exhibit 6.7: Difference between ELT and Matched Comparison Schools in Student Growth by 


Subject and Grade, 2010-11 



Actual 

Estimated 

Estimated 

Statistical 

Significance 


ELT Mean 

Non-ELT Mean 

Difference 

(p-value) 


ELA 


Grade 4 

55 

51 

4.310 

0.088 

Grade 5 

48 

48 

0.082 

0.972 

Grade 6 

49 

46 

2.980 

0.190 

Grade 7 

52 

51 

1.124 

0.669 

Grade 8 

55 

49 

5.588 

0.026* 


Math 


Grade 4 

54 

45 

8.712 

0.002** 

Grade 5 

46 

43 

2.901 

0.245 

Grade 6 

53 

49 

4.420 

0.139 

Grade 7 

50 

50 

-0.285 

0.903 

Grade 8 

56 

52 

3.735 

0.127 


EXHIBIT READS: The estimated difference between the average student growth percentile of grade 4 students in ELT 
schools and matched comparison schools is 4.31 percentile points. The difference is not statistically significant. 

Notes: Statistical tests comparing means include dummy variables indicating which matched pair groupings and account for 
the fact that students are clustered within schools. 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE. 

Sample: Current ELT schools as of the 2010-11 school year and their matched comparison schools. 
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Alternative MCAS Metrics: Student Growth Percentiles for Study Schools Relative to All Schools 
in the State 

Exhibits 6.8 and 6.9 display ELT and matched comparison school SGPs relative to other schools in the 
state. Unlike Exhibit 6.6, which presents median SGPs for ELT and matched comparison schools, 

Exhibits 6.7 and 6.8 show the full distribution of the percentiles and compare the study schools’ 
distributions to those statewide. Exhibit 6.8 presents the SGP distributions broken into five groups of 
growth (very low, low, moderate, high, and very high). MA ESE developed these growth categories such 
that statewide, each group contains 20 percent of all students. The “very low” category indicates students 
below the 20 th percentile for student growth, “low” indicates students between the 20 th to 40 th percentile, 
“moderate” indicates students between the 40 th to 60 th percentile, “high” indicates students between the 
60 th to 80 th percentile, and “very high” indicates students at or above the 80 th percentile in student growth. 

Exhibit 6.8 displays the percentage of students in ELT and matched comparison schools that fall into each 
of the growth categories for the 2010-11 school year (and comparing growth against data from 2009-10 
and 2010-1 1). 69 The graphs illustrate that the SGP distributions for the study sample — both ELT and 
matched comparison schools — were similar to the statewide distributions in both ELA (Panel 1) and math 
(Panel 2). For example, on the 201 1 ELA portion of the MCAS, 22 percent of students in ELT schools 
had a SGP that was greater than the 80 th percentile (meaning these students scored higher than 80 percent 
of their peers), while 19 percent of students in matched comparison schools achieved SGPs that were 
above the 80' 1 ’ percentile. To compare these to the statewide results, recall that, by design, 20 percent of 
students statewide had a SGP that was greater than the 80 th percentile. Comparing ELT and matched 
comparison schools, the exhibit suggests that the main difference between groups appeal's for math (Panel 
2), where 38 percent of students in ELT schools fall into the very low and low categories while 44 percent 
of students in matched comparison schools have student growth percentiles in the very low and low 
categories. Together, examining the full distribution of the SGPs reveals that differences between ELT 
and matched comparison schools may be smaller than suggested when looking solely at a small portion of 
the distribution, such as the median scores presented in Exhibit 6.6. 

Exhibit 6.9 collapses the five categories presented above (very low, low, moderate, high, and very high) 
into three, corresponding to high (SGP greater than 60 lh percentile), middle (SGP between the 40 lh and 
60 th percentiles), and low (SGP below the 40 th percentile) median student growth percentiles, and presents 
data for four consecutive school years. These descriptive results illustrate that the study schools — both 
ELT and matched comparison schools — are clustered in the middle category, again consistent with the 
findings presented above. However, due to the limitations of the metric noted previously in this section, 
including limited data availability and the method of calculating the scores, these analyses cannot take 
into account the statistical controls reflected in the more robust impact analyses presented in the next 
section. 


69 See http://www.doe.mass.edu/mcas/growth/InterpretiveGuide.pdf for additional information. 
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Exhibit 6.8: Student Growth Percentile Distributions in ELT and Matched Comparison Schools, 
2010-11 

Panel 1: ELA 



Percent of Students 


□ Very Low DLow DModerate DHigh uVery High 


Panel 2: Math 



Percent of Students 


□ Very Low DLow DModerate DHigh dVery High 


EXHIBIT READS: In spring 201 1 , 20 percent of ELT students and 22 percent of matched comparison students had ELA 
student growth percentiles that were below the 20th percentile on the ELA portion of the MCAS. 

Note: The “very low” category indicates students below the 20 th percentile for student growth, “low” indicates students 
between the 20 lh to 40 th percentile, “moderate” indicates students between the 40 th to 60 lh percentile, “high" indicates students 
between the 60 th to 80 lh percentile, and “very high” indicates students at or above the 80 lh percentile in student growth. 

Source: Individual student records obtained from MA ESE 

Sample: Current ELT schools as of the 2010-11 school year and their matched comparison schools. 
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Exhibit 6.9: Number of ELT and Matched Comparison Schools with High, Middle, and Low 
Median Student Growth Percentiles, 2008-11 

Panel 1: ELA 
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ELT MC 

ELT MC 

2008 

2009 

2010 

2011 


□ High Median SGP (>60th Percentile) 

■ Middle Median SGP (40th-60th Percentile) 

□ Low Median SGP (<40th Percentile) 


EXHIBIT READS: In 2008 for ELA, 3 ELT schools had a low median student growth percentile (SGP), 1 1 had a mid-range 
median SGP, and 1 had a high median SGP. There were no matched comparison schools with a low median SGP, 12 with a 
mid-range median SGP, and 2 with a high median SGP. 

Note: By design, the overall state median student growth percentile is 50, in all years and subjects. 

Source: School-level data tables downloaded from the MA ESE website ( http://profiles.doe.mass.edu/ ) 

Sample: Current ELT schools as of the 2010-11 school year, and their matched comparison schools. Data for 2008 only 
includes cohorts 1 and 2 of ELT and matched comparison schools. 
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Effect of ELT on MCAS Scores 

Descriptive analyses presented above indicate substantial variation in the pre- and post-ELT MCAS 
scores for ELT schools. In addition, there was considerable variation in the change between pre- and post- 
ELT MCAS scores among both groups of schools. However, these analyses omitted statistical controls 
important for identifying any impact of ELT on achievement outcomes. As such, the following section 
presents estimated effects of ELT on MCAS scores across cohorts after one, two, three and four years of 
implementation (Exhibit 6.10). Recall that these models compare data from ELT and matched comparison 
schools prior to and after the implementation of ELT, and the models use student-level covariates and 
school and year fixed effects to control for a number of plausible alternative explanations for the 
estimated effects. The effect of ELT is modeled using five variables: (1) whether the outcome is from an 
ELT school in its first year of implementation (and is coded zero otherwise); (2) whether the outcome is 
from an ELT school in its second year of implementation; (3) whether the outcome is from an ELT school 
in the third year of implementation; (4) whether the outcome is from an ELT school in its fourth year of 
implementation; and (5) whether the outcome is from an ELT school in the fifth year of implementation. 
These variables provide separate estimates of the effects of ELT across cohorts during their first, second, 
third, fourth, and fifth year of implementation over and above any changes in the outcomes. Further, the 
models appropriately account for clustering of students in schools using the cluster-robust variance 
estimator (also known as the “sandwich” standard errors; White, 1984 and Liang and Zeger, 1986). 

For ease of interpretation and analysis, the impact estimates are presented as z-scores, which can be 
interpreted as the student’s score relative to the statewide average. That is, the z-score is calculated by 
subtracting the average MCAS raw score for all the students in that grade and year in the state from the 
student’s raw MCAS score, and dividing by the standard deviation in that grade and year. Therefore, a 
student who scored at the average in the state for that year would have a z-score of zero. A positive score 
would indicate that the student scored above the state average, and a negative score that the student scored 
below the statewide average. Using z-scores as the metric allows for the estimated effect of ELT to be 
interpreted as an effect size, or a proportion of a standard deviation. 

Exhibit 6.10 displays the estimated effect of ELT on MCAS scores after one, two, three, and four years of 
ELT implementation. The y-axes indicate the size of the effect of ELT expressed in effect sizes, or 
standard deviation units. Therefore, bars that appear above the line labeled “0” indicate positive effects of 
ELT, indicating that ELT students scored higher than estimates of how they would have performed in the 
absence of ELT on those tests. Bars that appear below the line indicate negative effects of ELT, such that 
the scores for ELT students are lower than estimates of how these students would have performed in the 
absence of ELT. Results are presented by grade, as seen across the x-axes. The light blue bars correspond 
to ELA scores, the purple bars correspond to math scores, and the yellow bars indicate science scores. 

The charts indicate that there was only one case in which ELT had a significant impact on MCAS scores. 
After four years of implementation (in Panel 4 of Exhibit 6.10), students in ELT schools perform 
significantly higher in grade 5 science than would be estimated in the absence of ELT. Specifically, the 
effect size indicates that students in ELT schools are estimated to score .28 standard deviations (or 
approximately 2.7 points) higher on the 5 th grade science MCAS test than they would have in the absence 
of ELT. 70 Yet, because the analysis tests hypotheses about eight outcomes over four separate years of 
ELT implementation, it is likely that at least one effect would appear statistically significant solely due to 


70 


The 2010-11 statewide science grade 5 mean was 36.21 and the statewide standard deviation was 9.66. 
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chance. 71 There were no statistically significant impacts of ELT on ELA and math or in grade 8 science 
scores across any of the first four years of ELT implementation (see Appendix F for more detail on 
findings after five years of implementation). 

Some of the effect sizes presented in Exhibit 6.10 seem to become more positive over time. To explore 
whether cumulative changes over time were systematically related to ELT status, the study team 
estimated an alternative specification that models the ELT effect as an immediate jump (or fall) following 
the program’s implementation in a school and a linear annual change thereafter in the difference between 
the test scores of the ELT school and its matched comparison school. Note that this alternative 
specification can be regarded as more restrictive than the primary analytic model specification, as it 
assumes the annual change in the ELT effect is constant. The resulting estimates indicate there is no 
statistically significant trend in the impacts of ELT over time for any of the eight outcomes tested. 

Finally, it should be noted that in addition to the academic outcome models described above, Appendix F 
includes a number of specification and sensitivity models that test whether the results of the student 
achievement analysis would have differed had different analytic decisions been made. The specific 
scenarios tested include the following: 

• Tests using an alternative specification of the model, which included pre-ELT implementation 
group trends (ELT versus matched comparison) in student performance; 

• A test that included all ELT schools in all implementation years, rather than excluding schools 
from the analyses after they exited the sample; and 

• Tests reducing the sample to only five and only three years of pre-ELT implementation data for 
each cohort, using both the primary model and the alternative specification with pre-ELT group 
trends. 

Results, of these alternative specification models, however, do not change the conclusions drawn from the 
primary analytic model that finds no significant effect of ELT on the majority of the MCAS outcomes 
tested. 


71 Statistically, one in twenty (representing 5 percent) of the effects is likely to appear statistically significant due 
to chance when conducting hypothesis testing with the 5 percent critical significance value. In the case of the 
effect of ELT on MCAS scores, the study tested 32 ELT effects (8 outcomes x 4 years of ELT implementation 
= 32 potential ELT effects) and, thus, would expect to find between one and two incorrectly identified 
significant effects. 
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Exhibit 6.10: Effect of ELT on MCAS Subject/Grade Tests across Cohorts, by Implementation Year 
Panel 1: Effect of ELT on MCAS Scores after One Year of Implementation 
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Panel 2: Effect of ELT on MCAS Scores after Two Years of Implementation 
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Exhibit 6.10: Effect of ELT on MCAS Subject/Grade Tests across Cohorts, by Implementation Year 
Panel 3: Effect of ELT on MCAS Scores after Three Years of Implementation 
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Panel 4: Effect of ELT on MCAS Scores after Four Years of Implementation 
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EXHIBIT READS: After the first year of ELT implementation, students in ELT schools, on average, score 0.07 standard 
deviations below what they were predicted to score in the absence of ELT on the 3 ld grade ELA portion of the MCAS test. This 
difference was not statistically significant. 

Note: Full tables of these findings are presented appendix table F.3. Results are not shown for 8th grade science after one year 
because the estimated difference was 0.00. 

* statistically significant at p < .05 level 

** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE. 

Sample: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-2011. The sample sizes 
vary by cohort, implementation year, subject, and grade. The minimum number of schools is 29 and the maximum number of 
schools is 38. The minimum number of test records is 17,293 and the maximum number of test records is 39,242. 
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ELT Schools in Context: Comparisons to Other Schools in the District and State 

The analyses presented above indicate that there is a significant effect of ELT on 5 th grade science scores, 
but that other effects are not significant. A logical follow-on question is how the study sample fares 
relative to other types of schools — those in the host districts and those across the state as a whole. It 
should be noted, however, that although these alternative comparison groups are intuitively attractive, any 
comparisons must be approached with caution, as they are less able to control fully for factors that could 
bias the ELT impact estimates. That is, if another program systematically affected student achievement in 
the ELT schools differently than it affected other schools in the district or state, these models would 
misattribute this effect (positive or negative) to the ELT program. As a result, although comparisons to 
schools in ELT host districts and the state may identify significant differences between the study schools 
and other groups of schools, such comparisons are less rigorously able to attribute those differences solely 
to ELT implementation. 

The results of comparing ELT schools to other schools in the host districts and to other schools in the 
state are presented in Exhibit 6. 1 1. The underlying models are identical to those reflected in Exhibit 6. 10 
that use matched comparison schools, and that leverage pre-ELT data to control for observable and 
unobservable, stable characteristics of schools. These results indicate a few statistically significant 
differences that are consistent across these two new comparison groups. Comparing the ELT schools to 
all schools in the host districts as well as to all schools in the state, after two years of implementation, 3 rd 
grade students in ELT schools scored significantly lower on the ELA MCAS test. However after four 
years of implementation, 6 th grade students in ELT schools scored significantly higher on the math MCAS 
test, and 5 th and 8 th grade students scored significantly higher on the science MCAS tests. Additionally, 
after one year of implementation, 7 th grade students scored significantly higher on the ELA test than 
students in non-ELT schools across the state, and after three years of implementation, 6 th grade students in 
ELT schools scored significantly higher on the math test than students in non-ELT schools across the 
state. No other differences were statistically significant; thus any insignificant positive (or negative) 
coefficients were indistinguishable from zero. 
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Exhibit 6.11: Effect of ELT on MCAS Subject/Grade Tests across Cohorts, by Implementation 
Year, Using Alternative Samples of Comparison Schools 
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Exhibit 6.11: Effect of ELT on MCAS Subject/Grade Tests across Cohorts, by Implementation 
Year, Using Alternative Samples of Comparison Schools 
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Grade 8 

0.10 
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0.19* 

0.025 
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EXHIBIT READS: In the first year of ELT implementation, the estimated effect of ELT on 3 ld grade ELA scores when using 
the matched comparison schools chosen for the study as the comparison group was -0.07 standard deviations, which was not 
statistically significant. Using instead all other schools in ELT host districts or all non-ELT schools statewide as comparison 
groups, the estimated effects were -0.08 and -0.09 standard deviations respectively, neither of which were statistically 
significant. 

Note: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 
Source: Individual student records obtained from MA ESE. 

Sample: Student MCAS records from all schools in state from 2001-2002 to 2010-2011. The sample sizes vary by the group of 
comparison schools. 


Exploring the Relationship between implementation and Achievement 

The level of ELT implementation is another component of the initiative important to consider in relation 
to academic achievement outcomes. Specifically, one might hypothesize that those ELT schools that have 
more fully implemented ELT components (e.g., time for enrichment, opportunities for teacher 
collaborative planning) may have MCAS test scores that reflect such implementation. To address this 
possibility, the study conducted exploratory analyses of the relationship between student achievement and 
levels of implementation to begin to address the question about whether variation in implementation 
might help explain variation in student achievement. The first set of analyses explores the link between 
implementation fidelity and student achievement descriptively by plotting scores on the implementation 
index against proficiency scores for each grade and subject combination (see Exhibit 6.12). The second 
set of analyses investigates the relationship between implementation fidelity and ELT impacts using the 
short interrupted time series with comparison group design used in the previous section (see Exhibit 
6.13). 

It is important to note that these results are exploratory, as although the implementation index (discussed 
in greater detail in Chapter 3) developed for this study addresses the types of activities ELT schools 
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engaged in and the amount of time ELT schools spent on various activities, it does not measure the 
quality of ELT activities. Further, it is important to note that schools may well choose to implement ELT 
(and/or other initiatives) in ways that are also related to student achievement. If, for example, higher 
implementing schools also had high levels of parental involvement at the onset of the study period and 
higher levels of parental involvement translates into improved student achievement, one could 
erroneously attribute any relationship between implementation fidelity and student achievement to ELT, 
when improved student achievement actually reflects increased parent involvement. This is particularly 
important to keep in mind when examining results of descriptive analyses that include no statistical 
controls. This issue is also relevant to the second set of analyses that employ the study’s short interrupted 
time series with a comparison group design, such that the impact differences between higher and lower 
implementing schools cannot be causally attributed to differences in implementation fidelity. 

Relationship between the Percent of a School’s Students Reaching the Proficient or Advanced 
MCAS Levels and ELT Implementation 

Exhibit 6.12 presents results of exploratory analyses linking scores on the implementation index with 
student achievement results. Each of the fourteen charts each present scores on the index relative to the 
percent of students, within a school, who have scored at proficient or advanced on a specific MCAS test 
in a particular grade. Panel 1 presents data for the elementary grades and Panel 2 presents data for the 
middle school grades. These results suggest no clear pattern or meaningful relationship between scores on 
the index and student achievement. To further investigate any potential relationship between 
implementation index scores and student achievement, analyses linking index scores to median SGPs 
were also conducted. The results of the SGP analyses (found in Appendix F) are similar to those 
displayed in Exhibit 6.12, and similarly illustrate no consistent pattern between implementation index 
scores and SGPs. 
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Exhibit 6.12: Percent of Students Scoring Proficient or Advanced, 2011 MCAS Scores and ELT Implementation Index Scores, ELT Schools 
Panel 1: Elementary Grades 
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EXHIBIT READS: The grade 3 ELA graph indicates, for each ELT school, the percentage of students scoring proficient or advanced on the 3 rd grade ELA portion of the MCAS test in 2010-11 (y-axis) 
and the school's total score on the implementation index developed for the study (x-axis). 

Note: Each point on each graph represents a single ELT school. 

Source: Individual student records obtained from MA ESE: Abt Associates' Interviews ofMA ELT School Principals, Spring 2011; Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011. 
Sample: Current ELT schools as of the 2010-1 1 school year. 
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Exhibit 6.12: Percent of Students Scoring Proficient or Advanced, 2011 MCAS Scores and ELT Implementation Index Scores, ELT Schools 
Panel 2: Middle School Grades 
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EXHIBIT READS: The grade 6 ELA graph indicates, for each ELT school, the percentage of students scoring proficient or advanced on the 6 lh grade ELA portion of the MCAS test in 2010-11 (y-axis) 
and the school's total score on the implementation index developed for the study (x-axis). 

Note: Each point on each graph represents a single ELT school. 

Source: Individual student records obtained from MA ESE; Abt Associates’ Interviews ofMA ELT School Principals, Spring 2011; Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011. 
Sample: Current ELT schools as of the 2010-11 school year. 


pg. 146 | Chapter 6. Effect of ELT on School Characteristics and Student Achievement 


Abt Associates Inc. 


ELT Evaluation Year 5 Integrated Report 


Heterogeneity of ELT Effect by Level of ELT Implementation in Schools 

It is possible that the effect of ELT differs for schools with higher or lower scores on the implementation 
index. To investigate the potential of such a heterogeneous ELT effect, ELT schools were divided into 
two implementation groups: higher implementing and lower implementing schools. The median 
implementation index score served as the cut-score, with schools scoring at or above the median (which 
equaled 11) designated as higher implementing schools and those schools scoring below the median 
implementation index score labeled the low implementing schools. 72 The models used in the previous 
student achievement sections were estimated separately for higher and lower implementing schools and 
their matched counterparts. The statistical significance of the resulting ELT effects are assessed and the 
estimated effects for the two groups were compared to determine if there were significant differences in 
the effect of ELT by implementation level. 73 

Exhibit 6.13 presents the average difference of the ELT effect between higher and lower implementing 
schools. The y-axes indicate the effect size of the estimated difference. Therefore, bars that appear above 
the line labeled “0” indicate that the effect of ELT in higher implementing schools is greater than the 
effect of ELT in lower implementing schools. Bars that appear below the line indicate that the effect of 
ELT in higher implementing schools is less than the effect of ELT in lower implementing schools. 

Results are presented by grade, as seen across the x-axes. The light blue bars correspond to ELA scores, 
the purple bars correspond to math scores, and the yellow bars indicate science scores. 

The only statistically significant differences in the effect of ELT between higher and lower implementing 
schools are for grade 8 math after three and four years of implementation (in Panels 3 and 4 of Exhibit 
6.13). Students in higher implementing ELT schools perform higher on the 8 lh grade math MCAS test 
than do students in lower implementing ELT schools. These results suggest that there is some 
heterogeneity in the effect of ELT by the level of ELT implementation in a school, but that the 
heterogeneity may be most relevant for math and 8 th grade students. The effect size indicates that students 
in higher implementing schools are estimated to score between 0.40 and 0.50 standard deviations greater 
on the 8 th grade math MCAS test than students in lower implementing schools after three and four years 
of ELT implementation. Findings after five years of implementation and the results of the individual 
models that estimated the effect of ELT in higher and lower implementing schools can be found in 
Appendix F. 


71 This cut-point resulted in 8 ELT higher implementing schools and 7 ELT lower implementing schools. 

73 Assignment of matched comparison schools to the higher or lower implementing grouping was determined by 
their ELT pair. For example, if a matched comparison school’s ELT school was designated a higher 
implementing school, that matched comparison school was included in the model estimating the effect of ELT 
on higher implementing schools. 
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Exhibit 6.13: Differences in the Effect of ELT between Higher and Lower Implementing Schools 

Panel 1 : After One Year of Implementation 
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Panel 2: After Two Years of Implementation 
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Exhibit 6.13: Differences in the Effect of ELT between Higher and Lower Implementing Schools 

Panel 3: After Three Years of Implementation 
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Panel 4: After Four Years of Implementation 
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EXHIBIT READS: After the first year of ELT implementation, the effect of ELT on grade 3 ELA MCAS scores in higher 
implementing schools, on average, was .16 standard deviations lower than the effect of ELT in lower implementing schools. This 
difference was not statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE; Abt Associates’ Interviews ofMA ELT and Matched Comparison 
School Principals, Spring 2011; Abt Associates’ Surveys ofMA ELT and Matched Comparison School Teachers, Spring 2011 
Sample: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-2011. Only ELT schools 
and their matched comparison schools with full data on the implementation index are included in these analyses. The sample 
sizes vary by cohort, implementation year, subject, and grade. The minimum number of schools is 15 and the maximum number 
of schools is 24. The minimum number of test records is 10,048 and the maximum number of test records is 27,828. 
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Summary 

Taken together, the descriptive and impact analyses described in this chapter yield remarkably consistent 
results. The study schools’ stability in terms of school characteristics and student mobility suggests that 
any observed differences could indeed be attributed to the ELT initiative. The descriptive data about 
levels of student performance suggest considerable variation across schools, yet few systematic patterns 
in performance levels at two time points. Additional descriptive analyses using student growth percentiles 
for ELT and matched comparison schools illustrate higher student growth in ELT schools. However, 
without accounting for pre-ELT data, school and year effects and the full set of post-ELT data, these 
differences cannot be attributed to ELT. 

The more rigorous analyses indicate a significant impact of ELT on student performance for one grade — 
5 th — and one subject within that grade, science, taking pre-ELT data, school and year effects, and post- 
ELT data into account. Statistical comparisons of ELT schools to other (non-ELT) schools in their host 
districts and the state as a whole also indicate few meaningful differences. Finally, exploratory analyses 
linking ELT implementation to student outcomes do not provide clear evidence of patterns or 
associations. There is, however, some evidence of a heterogeneous ELT effect, such that the effect of 
ELT on grade 8 math is more positive in higher implementing schools than in lower implementing 
schools after three and four years of implementation. Otherwise, the level of ELT implementation in 
schools does not appear to differentially affect student achievement. 

As measured by this study, ELT implementation alone does not appear to explain why ELT has minimal 
effects on student achievement. It is important to note that that the implementation index does not 
measure the quality of ELT implementation; it may well be that it is the quality of ELT (and other 
instructional activities) that matters most for improving student outcomes. The fact that some matched 
comparison study schools implemented ELT-like components also serves to blur the potential treatment 
contrast between the two groups of schools. To address this last issue, the evaluation team estimated a set 
of models that control for the differences in ELT implementation between each ELT-matched comparison 
school pair, and the results were consistent with the results from the primary impact model described 
above. 

Another potential explanation for minimal evidence of impact could be limited statistical power to detect 
effects. The minimal detectable effect size (MDES) in the majority of impact models is .25, which is 
sufficient to detect meaningful changes in student achievement. The models with the greatest number of 
schools (e.g., the number of schools included after one and two years of ELT implementation — before 
some schools exited the initiative) have greater statistical power. The MDES does grow when estimating 
the effects after three and four years of implementation, and is largest for ELA in grade 3 and 4 and for 
Math in grade 8. Nevertheless, it is important to note that the estimated ELT effects for these outcomes 
are generally small (0.10 standard deviations or less), which suggests that it is not statistical power that is 
behind the lack of significant findings (e.g., even had an alternative model specification been employed 
that pooled grades within each subject, the estimated effect would still be unlikely to reach statistical 
significance). 
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Chapter 7. Conclusions and Recommendations 


ELT schools continued efforts to implement the core ELT components, although ongoing examination of 
the implementation of the initiative revealed considerable variation both in the extent and manner of 
schools’ implementation of ELT. Over the five years of the initiative, the number of participating schools, 
as well as school, district, and state contexts for ELT have changed. So too have the expectations of the 
participating schools. Among the contextual factors that have likely affected ELT implementation in the 
2010-201 1 year include: ESE’s use of performance agreements and monitoring activities; the design and 
use of technical assistance to schools; and the prevalence of ELT-like components as a school reform 
strategy throughout the Commonwealth. Each is discussed below. 

Performance Agreements and Monitoring. In the 2009-10 school year, ESE launched multi-year 
performance agreements outlining specific benchmarks for ELT schools. The performance agreement 
template can be found in Appendix A. These agreements included more explicit and detailed expectations 
about ELT implementation, and schools were informed that they could be discontinued from the ELT 
initiative pending their adherence to the performance agreements. 

Technical Assistance. Mass 2020 and Focus on Results continued their efforts to provide customized 
technical assistance approaches and each reported ongoing efforts to support schools more 
comprehensively on some core components. 

Prevalence of Increased Time. Large-scale federal initiatives such as Race to the Top and School 
Improvement Grant (SIG) funding each include use of increased learning time as one mechanism schools 
and districts can use to improve outcomes for students in the lowest-performing schools. In the 
Commonwealth, for example, ESE identified 34 of the state’s lowest-performing schools as Level 4 
schools; these schools could adopt increased time as one part of a school turnaround strategy, and many 
of the Level 4 schools have elected to expand their schedules. However, it should be noted that just one 
Level 4 school is participating in the ELT initiative that is the focus of this research. Massachusetts’ Race 
to the Top efforts also includes the use of increased learning time as a school turnaround strategy for both 
districts and schools to consider. 

Recent media attention 74 has also highlighted attention on ELT in Massachusetts in particular. So too did 
the October 2011 ELT Summit convened by the National Center on Time and Learning, which provided 
information about ELT implementation to district and school officials from across the country as well as 
to those within Massachusetts. 


Summary of Implementation of ELT 

The study collected more systematic data about the availability and allocation of time in 2010-11. ELT 
implementation begins with a basic commodity: increased time for learning. The ELT schools were all 
charged with increasing the amount of time in the school year by 300 hours (or more) each year, and all 
of the ELT schools reported having more time in their schedules. The ELT school day was just under 
eight hours, on average, in 2010-11, and the largest block of time (almost five hours, on average) was 


74 See Boston Globe editorial: http://bostonglobe.com/opinion/editorials/201 1/12/15/leadership-and-flexibility- 
not-buses-improve-schools/YeYr6QzBkDRQFawYv40U4J/story.html 
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allocated to core academics. More specifically, the average amount of time (daily) was allocated as 
follows in ELT schools: 

• Of the almost five hours allocated to core academics: 

- The amount of time scheduled for English Language Arts (ELA) was 1 hour and 45 minutes, 
on average. 

- Nearly 90 minutes were allocated to math instruction, on average. 

- An average of nearly 1 hour was scheduled for science and 45 minutes to social studies per 
day. 


Not surprisingly, time allocations for core subjects varied somewhat by grade. Specifically, an average of 
about 45 minutes more each day was allocated to ELA in 5 lh than in 8 th grade. Slightly more time (12 
minutes, on average) was allocated to math in 5 th than in 8 th grade. About 20 fewer minutes were allocated 
to science and 15 minutes fewer to social studies in 5 th than 8 th grade, on average. 

As in past years, individual schools varied considerably in how they allocated time to various 
instructional activities. Consistent with the time allocations, schools’ academic focus areas were related to 
ELA more often than other subject areas. Additionally, administrator and staff perceptions were 
remarkably congruent about their school’s respective academic focus. Most teachers and principals also 
agreed that instructional practice at the school was informed by the academic focus, and that data were 
used to adjust instructional practices as needed. 

One key tenet of ELT is that students have the opportunity to participate in activities designed to enhance 
and enrich their learning. How schools operationalized enrichment activities, however, varied 
considerably across ELT schools; enrichment was variously defined as “electives,” “specials,” both 
academic and non-academic, and as integrated or appended to the school day. Regardless of definitions, 
most teachers in ELT schools reported that all students participated in at least some enrichment activities, 
and most teachers also indicated that enrichment activities occurred frequently enough to be valuable and 
were well-integrated into the school schedule. At most schools, enrichment activities were taught by a 
combination of classroom teachers, other school staff, and community partner staff. Those ELT schools 
serving middle grade students appeared to allocate more time than those serving predominantly 
elementary grades for both academic and non-academic enrichment. 

Successful implementation of ELT is also hypothesized to require more deliberate coordination and 
planning on the part of school and instructional staff, often through the use of designated blocks of time 
for teachers to engage in collaborative planning about instruction. In 2010-1 1, more than half of ELT 
teachers (65 percent) participated in collaborative planning time weekly or more often. Only a small 
proportion of teachers (16 percent) reported never having participated in collaborative planning. 


Teachers reported participating in multiple activities during collaborative planning time, including 
analyzing student data, strategizing about instructional practices, and/or reviewing student work. The 
majority of teachers who reported participating in an activity also reported that the activity was useful. 

Overall, half of teachers reported having completed more than 26 hours of professional development over 
the course of the school year, 12 percent reported completing between 51 and 75 hours of professional 
development, and 1 1 percent completed more than 75 hours of professional development. More than 
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three-quarters of ELT teachers (77 percent) agreed that in-school professional development contributed to 
the development of a professional learning community of teachers. 


Describing Variation in Schools’ Implementation of ELT 

Over the past five years, as successive cohorts of ELT schools have launched and continued their 
implementation efforts, data from study-administered individual and group interviews, and surveys, 
coupled with information from ESE and Mass 2020, have highlighted the variability of schools’ level of 
implementation. In the 2010-11 school year, the study examined variation in implementation using an 
updated implementation metric first developed for use with 2009-10 data. The index incorporates 
information from principal interviews about time allocation across multiple academic, enrichment, and 
support instruction/activities, as well as information from teacher and student survey respondents about 
resources, supports, leadership, and perceptions of school climate. The index categorizes schools’ 
implementation levels on various dimensions corresponding to hypothesized dimensions of effective ELT 
implementation; it allows the study team to array each school’s collective interview and survey responses 
into multi-dimensional metrics. The index also allows researchers to examine both school-specific and 
initiative- wide patterns of implementation. Finally, because many of the principles of effective ELT 
implementation could apply equally well to more generic or other specific school improvement initiatives, 
the index was designed to apply to both ELT and matched comparison schools. 

The results from applying the index indicate, not surprisingly, considerable variation. As a group, the 
ELT schools appeared to be implementing and sustaining activities across multiple core components of 
ELT. As a set of individual schools, there was substantial variation in summative scores on the 
implementation index as well as scores on individual criteria. The ELT schools had consistently higher 
scores than did comparison schools on six of eight criteria, and had higher total scores, on average. Many 
comparison schools also appeared to be implementing at least some components considered core 
expectations of the ELT initiative, in particular, components focusing on general leadership and school- 
wide academic foci. As was the case with 2009-10 patterns of results, ELT schools’ implementation index 
scores were consistently higher on the criteria related to enrichment. 


Describing Effects of ELT 

This report examined the differences between ELT and matched comparison schools, on average, as well 
as across cohorts and school configurations. The findings based on comparisons between the two types of 
schools are presented below. Findings related to non-academic outcomes are presented first, then 
academic outcomes, followed by a brief discussion of factors that may have influenced the results. 

Effects of ELT on Non-Academic Outcomes 

The overall outcome findings include few statistically significant and practically meaningful differences 
in outcomes for schools, students, and teachers between ELT and matched comparison schools. These 
patterns are across a range of findings related to school-wide characteristics, time use, students’ 
achievement outcomes, student behaviors, and student and teacher attitudes. 

The school day was longer, on average in ELT than matched comparison schools, and ELT schools 
allocated significantly more time to core academic subjects than matched comparison schools. Not 
surprisingly, ELT schools also were able to allocate more time to the various instructional activities that 
occurred during the day. Specifically, ELT schools across both 5 th and 8 th grade allocated significantly 
more time to ELA than matched comparison schools, more time for 5 th grade math classes, more time for 
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8 th grade science classes, more time for 5 th grade specials, and more time for academic support periods, 
enrichment, and other activities than likely would have occurred in the absence of ELT. 

There were few meaningful differences on teacher outcomes as measured by study-developed surveys. 
While ELT teachers reported being more satisfied with the amounts of time available for instruction in 
ELA, math, science, and social studies, they were also more likely to report that teacher and staff fatigue, 
as well as student fatigue, were problem areas; these results are similar to results from earlier years. 

There were also not many statistically significant and meaningful impacts of ELT on 5 th and 8 th grade 
students’ perspectives, as assessed by study surveys, although student responses from only two grade 
levels should not be assumed to represent the perceptions of all students in study schools. A higher 
proportion of ELT student indicated that they were tired in school. 

After at least four years of implementation, there were some impacts of ELT on students’ behavioral 
outcomes, assessed using state -provided SIMS data, although the magnitudes of observed differences are 
likely too small to have any educational or practical significance. 

Effects of ELT on Academic Outcomes 

There were no statistically significant effects of ELT after one, two, three, or four years of 
implementation on MCAS student achievement test outcomes for 3 rd , 4 th , or 7 th grade ELA, 4 th , 6 th , or 8 th 
grade math, or 8 th grade science. There was a statistically significant positive effect of ELT after four 
years of implementation on the MCAS 5 th grade science test. 

Descriptive Analyses 

As described above, there are few overall impacts of ELT on student achievement, as measured by MCAS 
scores. The impact analyses provide data on the impact of ELT on schools on average, and therefore may 
mask variation across individual schools’ academic achievement levels. This report includes descriptive 
analyses of data on students’ academic achievement at the individual ELT school level, to illustrate 
variation across the participating schools and that the average achievement levels vary considerably. 
Taken as a group, the schools’ mean achievement levels — across multiple grade levels and subject 
areas — do not appear to have changed much between the pre-implementation and 2010-11 school year. 

Descriptive analyses also compared ELT schools to schools in their respective districts and the state. 
Similar to the matched comparison school models, the results indicate few statistically significant 
differences. These analyses also highlight the fact that the ELT initiative as a whole includes schools 
serving underperforming students. 

Descriptive analyses linking ELT implementation to MCAS outcomes examined variation in scores by 
school using a number of metrics. Descriptive analyses restricted to ELT schools indicated variation in 
student performance levels among schools both before implementation began and in the most recent 
school year (2010-11), and indicated no consistent patterns of results. Additional descriptive analyses 
linking both ELT and matched comparison school to student growth percentiles suggested that ELT 
schools outperformed their matched comparison schools in 2011 on all MCAS tests in each of the grades. 
However, because these analyses do not include the robust statistical controls included in the impact 
models, differences cannot be attributed to ELT. 
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Discussion 

Across findings from interviews, surveys, and achievement data, the following themes seem clear: 

• There continues to be substantial variation across ELT schools’ level and approach to 
implementation (as measured by interviews, surveys and the study’s implementation index). 

• Measuring different aspects of time use is challenging: collecting information on a prototypical 
student in a given grade level may or may not reveal how students are supported by the ELT 
initiative and definitions of various activities/time uses are not consistent across schools. 

• There are some, but not many, differences — even descriptively — between ELT and comparison 
schools on survey and achievement outcomes. 

• The school reform landscape is dynamic and each successive year, more schools (outside of this 
ELT initiative) appear to be expanding the amount of time in their school year, and implementing 
reforms consistent with the core ELT components. 

Furthermore, the student achievement measures used to examine outcomes in this evaluation are broad 
proxies of academic knowledge. There may be other outcomes, both academic and non-academic, that 
could be assessed to describe both current and potential longer-term effects of ELT on both student 
learning and other important aspects of their lives (e.g., on-track high school performance for students 
who were in ELT schools for the duration of middle school relative to students in non-ELT schools, 
additional information on future educational and/or career aspirations). 

It is clear that after five years, the ELT initiative is a maturing initiative, although some participating 
schools were in their third year of implementation during 2010-11. Given the considerable variability in 
ELT implementation across schools, the considerable variation in schools’ initial motivation to become 
ELT schools, and ongoing changes in the education reform landscape during the same period, it may not 
be surprising that the study has yet to find significant student achievement gains attributable to this 
initiative. Expansion of time for learning is increasingly perceived as an important vehicle for improving 
educational outcomes for disadvantaged students in Massachusetts and elsewhere — if the additional time 
is used well. This study was able to assess the quantity and allocation of time, but did not measure the 
quality of instruction, enrichment, and other activities made possible by the additional time, and clearly, 
the quality of such activities is also important. 


Future Research 

There are a number of possible directions for future research focused on ELT, including but certainly not 
limited to exploring in even more depth how to measure and understand how time is used, assessing the 
relative effectiveness of various approaches to expanding the day, further understanding ELT’s cost- 
effectiveness, and examining outcomes other than student achievement. In addition, more understanding 
is needed about what higher performing ELT schools are doing differently from other schools in terms of 
program implementation, as well as the relationship between teacher quality and/or instructional quality 
and how time is used. It would also be interesting to further examine why teachers are generally positive 
about the opportunities afforded with additional time yet students are more negative about their 
experiences. Given the limited rigorous research evidence about both the effects of ELT on academic and 
other student outcomes, and on the specific mechanisms/factors that would help to understand whether, 
how, and for whom ELT is effective, the need for additional credible evidence is clear. 
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Exhibit A.1: 


ESE ELT Performance Agreement Template 



Massachusetts Department at 

ELEMENTARY & SECONDARY 

EDUCATION 


Massachusetts Expanded Learning Time to Support Student Success 
Performance Agreement Template 
(Produced in collaboration with Massachusetts 2020) 


Name of ELT School: 


1. USING EXPANDED TIME TO 
IMPROVE ACADEMIC OUTCOMES 

Goal: Our school will use additional time to accelerate student learning in core academic 
subjects by making meaningful improvements to the quality of instruction in support of 
school-wide achievement goals (Expectation II) 

Objective 1 : Ail students will make strong continual progress toward proficiency and excellence in 
reading and writing 

Measure 1 (MCAS): 

Measure 2 (Internal): 

Measure 3: 

Objective 2: All students will make strong continual progress toward proficiency and excellence in 
mathematics. 

Measure 1 (MCAS): 

Measure 2 (Internal): 

Measure 3: 

Objective 3: (determined by school; can be subject/skill specific, achievement gap, etc.) 

Measure 1: 

Measure 2: 

Measure 3: 
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2. USING EXPANDED TIME TO 
IMPROVE TEACHER LEADERSHIP & COLLABORATION 


Goal: Our school will effectively use additional time to build a professional culture of 
teacher leadership and collaboration (e.g. designated collaborative planning time, on-site 
targeted professional development) focused on strengthening instructional practice and 
meeting school-wide achievement goals (Expectation IV) 

Objective I : Teacher teams meet regularly to discuss and analyze student work and data for the 
purpose of setting high standards, assessing student learning, and strengthening the use of school-wide 
instructional practices. 

Measure I: 


Measure 2: 


Objective 2: Leadership team supports a community where faculty, staff, and administrators regularly 
engage in professional development, and data-driven analysis and planning to improve student learning. 

Measure I: 


Measure 2: 


Objective 3: (determined by school; can pick from the ELT Indicators* * or generate one that is school- 
specific) 

Measure I; 


Measure 2: 


*The ELT Indicators listed below would make strong objectives for this Goal: 

■ Time is used to engage staff in results-oriented discussion directly connected to curriculum, instruction, 
and assessment issues related to supporting all students in meeting state standards 

■ At least once annually, leadership team engages staff in review of the use of collaborative planning and 
professional development time to determine any necessary improvements 

■ Leadership team enables inter<lassroom visitations so teachers can observe peers and use information 
to improve instructional practice 

■ The school has created opportunities for teachers to meet on a regular basis for structured collaboration 
to analyze student data and strategize effective instructional practices and individual student needs 

• Leaders coordinate whole-school professional learning activities, including suggesting professional 

readings, providing opportunities for teacher-led professional development in key areas, and structuring 
opportunities for cross-team collaboration (Expectation V) 

■ Enrichment staff collaborate with core academic teachers to share lesson plans, content, individual 
student information (Expectation III) 
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3. USING EXPANDED TIME TO 
PROVIDE INTEGRATED ENRICHMENT OPPORTUNITIES 


Goal: Our school will effectively use additional time (in core and/or specialty classes) to 
offer enrichment opportunities that connect to state standards, build student skills and 
interests, and deepen student engagement in school/learning in support of school-wide 
achievement goals (Expectation III) 

Objective I: Enrichment opportunities help to develop students’ skills and talents, explore special 
interests, and perform/demonstrate/exhibit the products of their work. 

Measure I: 


Measure 2: 


Objective 2: Management is in place to support the integration of enrichment providers (both 
teachers and partners) and ensure alignment with school’s area of academic focus, culture, and 
operations. 

Measure I: 


Measure 2: 


Objective 3: (determined by school; can pick from the ELT Indicators* or generate one that is school- 
specific) 

Measure I: 


Measure 2: 


*The ELT Indicators listed below would make strong objectives for this Goal: 

■ Leadership team has developed an internal assessment system that monitors the impact of enrichment 
activities on student learning (Expectation III) 

■ Enrichment staff collaborate with core academic teachers to share lesson plans, content, individual 
student information (Expectation III) 

■ The allocation of resources maximizes individual attention for all students in academic focus area and in 
non-academic student support such as counseling, advisory groups, and student engagement in school 
(Expectation VI) 

■ Systems and safety nets are in place to help all students to overcome barriers to leaning and increase 
their engagement in learning (e.g. small group learning, advisory, counseling health and mental health 
support, etc.) (Expectation I) 

■ Students move beyond basic skills to master 21 st century skills (oral and written presentation, problem- 
solving teamwork, and use of technology) and employ across subject areas (Expectation II) 
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MASSACHUSETTS EXPANDED LEARNING TIME 
PERFORMANCE AGREEMENT APPROVALS 


For the Massachusetts Expanded Learning Time School: 


ELT School Name: 


Principal 

Signature: 


Principal 
Printed Name: 


Date: 



For the Massachusetts Expanded Learning Time District: 


ELT District Name: 


Superintendent 

Signature: 


Superintendent 
Printed Name: 


Date: 



For the Massachusetts Dept, of Elementary and Secondary Education: 


Commissioner/Designee 

Signature: 


Commissioner/Designee 
Printed Name: 


Date: 
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Exhibit A.2: ELT Expectations for Implementation 

Mass 2020 


t 

n i 


ELEMENTARY ft SECONDARY 

EDUCATION 


Expanded Learning Time 
Expectations for Implementation 


I. ELT Redesign Supports A Clear, School-wide Academic Focus 

The school's plan for implementation of ELT is aligned with the school's overall academic focus. 
This academic focus drives instructional improvement and continuous measurable growth in 
student learning throughout the redesigned day and year. The design and implementation of ELT 
is based on a data-driven assessment of student needs and works to support a clear set of school- 
wide achievement goals. 

II. Additional Time For Core Academics 

The school uses additional time in order to accelerate learning in core academic subjects by 
making meaningful improvements to the quality of instruction in support of school-wide 
achievement goals. 


III. Additional Time for Enrichment 

The school uses additional time (either in core and/or specialty classes) to offer enrichment 
opportunities that connect to state standards, build student skills and interests, and deepen 
student engagement in school/learning in support of school-wide achievement goals. 

IV. Additional Time for Teacher Leadership and Collaboration 

The school uses additional time to build a professional culture of teacher leadership and 
collaboration (e.g. designated collaborative planning time, on-site targeted professional 
development) focused on strengthening instructional practice and meeting school-wide 
achievement goals. 

V. Focused and Collaborative Leadership 

The principal as instructional leader and leadership team are fully committed to expanding 
learning time to improve instructional practice and to bringing many others - teachers, students, 
families, partners, and the community - into the process of redesign and implementation in 
support of school-wide achievement goals. 

VI. Resources are Aligned and Focused 

The school demonstrates clear evidence that it is making decisions around resource allocation 
(Time, People, Talent, Energy and Money) that are aligned with the successful implementation of 
the ELT redesign and focused on meeting school-wide achievement goals. 

VII. District Leadership Supports ELT 

The district actively supports all ELT schools around the ELT Expectations for Implementation by 
providing leadership, support, supervision, long-term district planning and creative problem 
solving to remove barriers and ensure schools can meet rigorous achievement goals. 


Focus on 

Results 


♦ 5901 Warner Ave., #441, Huntington Beach, CA 92649 

♦ 198 Tremont Street, #408, Boston, MA 02116 

♦ Phone: 888-743-1076 ♦ Fax: 508-869-0562 


Finding results-oriented solutions to the 
challenges of educating all students at high levels. 

www.focusonresults.net 
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Mass 2020 


Indicators of Implementation of the ELT Expectations 


1 e 

!) i 


Massachusetts Department ol 

BLEMENTARY & SECONDARY 

EDUCATION 


Expectation I. ELT Redesign Supports A Clear, School-wide Academic Focus 

The school's plan for implementation of ELT is aligned with the school's overall academic focus. This academic 
focus drives instructional improvement and continuous measurable growth in student learning throughout the 
redesigned day and year. The design and implementation of ELT is based on a data-driven assessment of student 
needs and works to support a clear set of school-wide achievement goals. 


PHASE II 

Evidence would include: 


PHASE I 

Evidence would include: 

• Principal and leadership team lead the staff in a critical 
examination of current student performance based on data 

• Tire entire staff has participated in determining a school- wide • 

instructional focus for their improvement efforts 

• School collects baseline data on student performance at the 

beginning of each school year and continuously monitors the • 

progress of all students toward goals 

• The adoption and regular use of at least two assessment measures -- • 

interim assessments that the school or district administers 
periodically during the year and the annual state-administered 
standardized assessment (MCAS) 

• Student performance data are posted to show improvement during 
the year and between years 

• The school systematically implements a program of interim 
assessments (4-6 times per year) in English language arts and 
mathematics that are aligned to school curriculum and state 
frameworks 

• Teachers use the data from the interim assessments to monitor and 
adjust instruction 


Resources, especially TIME, used strategically to support the 
instructional focus 

Most teachers engage in the collection and analysis of student 
performance data both for individual student’s progress and the progress 
of particular cohorts/groups of students 

Specific goals for improvement are set and data is publicly posted and 
showcased for the entire school community 

Systems and safety nets are in place to help all students to overcome 
barriers to learning and increase their engagement in learning (e.g. small 
group learning, advisory, counseling, health and mental health support, 
etc.) 
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Mass 2020 


Indicators of Implementation of the ELT Expectations 
Expectation II. Additional Time For Core Academics 


I) I 


Massachusetts Department ot 

ELEMENTARY & SECONDARY 

EDUCATION 


The school uses additional time in order to accelerate learning in core academic subjects by making meaningful 
improvements to the quality of instruction in support of school-wide achievement goals. 


PHASE I 


PHASE II 


Evidence would include: 


Evidence would include: 


• School leadership has begun to have meaningful discussions of what all 
students are capable of doing and expected to do in relation to core academic 
standards 

• The school implements state-aligned, standards -based curriculum to support 
student attainment of explicit, academically rigorous, performance standards 

• The school schedule for student learning provides adequate time for the 
delivery of instruction and provision of individualized support as needed in 
the core academic areas (English language arts, math, science, and social 
studies) 

• Changes in instructional practice to improve performance in core academic 
subjects have begun to be identified as a result of conversations about student 
work and data 

• School provides opportunities for students to receive supplemental instruction 
as necessary for skill development and acceleration by qualified staff 

• Students move beyond basic skills to master 21 st century skills (oral and 
written presentation, problem-solving, teamw'ork, and use of technology) and 
employ across subject areas 


• 3-5 school selected effective instructional practices 
implemented consistently in every classroom in support of 
the school focus 

• Principal and coaches provide supportive feedback through 
frequent classroom visitation, coaching, and modeling on the 
selected instructional practices 

• School-based professional development in selected 
instructional practices includes modeling and demonstrations 
by content coaches and visiting other classrooms in the 
school 

• The school has subject-area coaches, one each for English 
language arts/reading and one for mathematics, who are 
responsible to provide faculty with consistent classroom 
observation and feedback on the quality and effectiveness of 
curriculum delivery, instructional practice, and data use 
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Mass 2020 


Indicators of Implementation of the ELT Expectations 
Expectation III. Additional Time for Enrichment 


1 '■ 

i 


Massachusetts Department ot 

ELEMENTARY & SECONDARY 

EDUCATION 


The school uses additional time (either in core and/or specialty classes) to offer enrichment opportunities that 
connect to state standards, build student skills and interests, and deepen student engagement in school/learning 

in support of school-wide achievement goals. 


PHASE I 

Evidence would include: 

• School integrates enrichment opportunities into school day and year 
(including both content and use of personnel and other resources) 

• Enriclunent opportunities help to develop students’ talents, explore special 
interests, and perform/demonstrate/exhibit the products of their work on 
group and individual projects 

• School engages providers that provide enrichment activities (e.g. music, arts, 
drama, apprenticeships, etc.) that are connected to standards and engage 
students in active learning 

• Management in place to support the integration of enrichment providers 
(both teachers and partners) and ensure their alignment with school’s area of 
academic focus, culture, and operations 

• Leadership team creates expanded list of possible enriclunent partners to 
enhance cuniculum and student exposure to content 

• Classroom management practices consistent in all enriclunent classes (and 
consistent with core academic classes) 


PHASE II 

Evidence would include: 

• Enriclunent opportunities prepare students to engage in 
productive discourse, assume civic responsibility, participate in 
cooperative endeavors, and assume leadership roles 

• Goals/objectives for enriclunent classes aligned to curriculum in 
core academic classes 

• Enrichment activities are shared with families and exhibited in 
ways that increase family and community engagement in the 
school 

• Leadership team lias developed an internal assessment system 
that monitors the impact of enrichment activities on student 
learning 

• Enrichment staff collaborate with core academic teachers to 
share lesson plans, content, individual student information 

• Partners, when appropriate, enhance the type and quality of 
safety nets provided for all students, (e.g. the areas of health, 
mental health, social services, counseling, advisory, etc) 


Focus on 

Results 
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Indicators of Implementation of the ELT Expectations 


It 


Massachusetts Department of 

ELEMENTARY & SECONDARY 

EDUCATION 


Expectation IV. Additional Time for Teacher Leadership and Collaboration 

The school uses additional time to build a professional culture of teacher leadership and collaboration (e.g. 
designated collaborative planning time, on-site targeted professional development) focused on strengthening 
instructional practice and meeting school-wide achievement goals. 


PHASE I 

Evidence would include: 

• The school has created opportunities for teachers to meet on a regular 
basis for structured collaboration to analyze student data and 
strategize effective instructional practices and individual student 
needs 

• The leadership team, working with the principal, provides school- 
wide leadership around implementation of ELT 

• The leadership team has been trained in effective meeting strategies 
and distributed leadership 

• Inter-classroom visitation in search of potential effective instructional 
practices has been adopted school-wide in core academic subjects 

• Time is used to engage staff in results-oriented discussion directly 
connected to curriculum, instruction, and assessment issues related to 
supporting all students in meeting state standards 

• Agendas and minutes of meetings are maintained and are regularly 
shared with school leadership and the school community 

• Administrators, coaches, specialists, and interventionists actively 
participate in and support collaborative planning meetings 

• A variety of protocols are available for teachers to use during 
collaborative planning time in order to guide looking at student work, 
engaging in lesson study, building common assessments, and refining 
practice. 


PHASE II 

Evidence would include: 

• There is a collection of team meeting reports and schedules; these reports are used 
in school-wide decision-making 

• Teacher teams meet regularly to discuss and compare student work for the purpose 
of setting high standards, assessing their own and their students’ performance 

• Leadership team enables inter-classroom visitations so teachers can observe peers 
and use information to improve instructional practice 

• Professional development opportunities are linked directly to student and teacher 
needs. 

• Leadership team supports a community where faculty, staff', and administrators 
regularly engage in professional discourse, study and data-driven evaluation and 
planning to improve student learning, 

• Team meeting norms/protocols are continually revisited to ensure all team 
members operate as equals and demonstrate mutual respect. 

• Professional development activities are delivered primarily by teachers and leaders 
from within the school — along with outside experts, when necessary. 

• At least once annually, leadership team engages staff in review of the use of 
collaborative planning and professional development time to determine any 
necessary' improvements 
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Indicators of Implementation of the ELT Expectations 


I * 

n i 


Massachusetts Department at 

ELEMENTARY &. SECONDARY 

EDUCATION 


Expectation V. Focused and Collaborative Leadership 

The principal as instructional leader and leadership team are fully committed to expanding learning time to 
improve instructional practice and to bringing many others - teachers, students, families, school partners, and the 
community - into the process of redesign and implementation in support of school-wide achievement goals. 


PHASE I 

Evidence would include: 

• Principal supports the instructional focus through classroom visits, coaching, and 
modeling 

• Principal is an instructional leader, knowledgeable about current instructional 
practice in every classroom 

• The expanded school schedule allows the principal to meet with teacher teams to 
review student performance data and discuss adjustments to instruction based on 
the data 

• Principal and leadership team create opportunities to communicate and engage 
with students, families, school partners and the community in the process of 
redesign and implementation. 

• Principal visits all classrooms at least once per week and collects, analyzes, and 
uses the data to inform decision making about instructional leadership 

• Principal and leadership team clearly communicate how instructional practices 
connect to area of academic focus 

• Leaders coordinate whole-school professional learning activities, including 
suggesting professional readings, providing opportunities for teacher-led 
professional development in key areas, and structuring opportunities for cross- 
team collaboration 

• Families introduced to the redesigned, expanded school day and the school-wide 
academic focus 

• Principal and school leadership team hold themselves, school faculty and staff to 
high standards of professional practice and personal conduct 


PHASE II 

Evidence would include: 

• Principal and leadership team sets specific iastructional leadership 
goals that focus on monitoring and improving student learning 
throughout the year 

• Principal is a collaborative, supportive instructional leader committed 
to excellent teaching in every classroom 

• Principal uses data from classroom visits with leadership team to lead 
school in PD and allocating resources - especially time 

• Successes are celebrated regularly and frequently, and teams, 
individual teachers, and students are proudly upheld as exemplars 

• Teachers visit the classrooms of their colleagues on a regular basis and 
are use agreed-upon protocols 

• All family engagement activities (events, newsletters, conferences, 
etc.) support improving academic learning and broader student growth 

• A formal system of two-way communication and collaboration is in 
place with students, families, school partners and the community 
regarding design and implementation of the expanded learning time 
redesign. 
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Indicators of Implementation of the ELT Expectations 
Expectation VI. Resources are Aligned and Focused 


1 E 

R I 


Massachusetts Department of 

ELEMENTARY & SECONDARY 

EDUCATION 


The school demonstrates clear evidence that it is making decisions around resource allocation (Time, People, 
Talent, Energy and Money) that are aligned with the successful implementation of the ELT redesign and focused on 

meeting school-wide achievement goals. 


PHASE II 


Evidence would include: 


PHASE I 

Evidence would include: 

• Principal and leadership team evaluate resources from all 
sources (time, talent, dollars, and staff) to consider ways of 
more effectively supporting the instructional focus 

• The leadership team makes decisions regarding school day 
and school year schedules, the allocation of student learning 
time, position allocations, and discretionary spending 

• Principal and leadership team catalogue all programs and 
grants in the school and assess them in relation to 
instructional focus 

• Time has been allocated for regular meetings of the 
leadership team and teacher teams and for professional 
development during the school day 

• Teaching staff participates in investigating alternative uses of 
existing resources to better achieve school-wide academic 
goals 

• Principal and leadership team implement process to regularly 
share information regarding resource allocation with teachers 
and staff 


The resources at the school level (support staff, technology, time, and 
dollars) work together to support the school's instructional focus and the 
implementation of the ELT redesign 

Review of current plan shows reallocation of resources in support of the 
school's instructional focus when compared to previous plan 
A resource decision-making process that includes input from leadership 
team and a range of constituents aligns the organization, use of 
resources, and staffing to support student achievement in the 
instructional focus and the ELT redesign 

The allocation of resources maximizes individual attention for all 
students in the academic focus area and in non-academic student support 
such as counseling, advisory groups, and student engagement in school 
At the end of each school year, student data and resource allocation are 
reviewed and implementation plan is revised for new school year 
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Indicators of Implementation of the ELT Expectations 
Expectation VII. District Leadership Supports ELT 


1 H 

1 


Massachusetts Department of 

ELEMENTARY & SECONDARY 

EDUCATION 


The district actively supports all ELT schools around the ELT Expectations for Implementation by providing 
leadership, support, supervision, long-term district planning and creative problem solving to remove barriers and 

ensure schools can meet rigorous achievement goals. 


PHASE I 


PHASE II 


Evidence would include: 


Evidence would include: 


• District provides leadership in communicating the goals of the 
school’s ELT work within the broader school community 

• District has identified a person in the central office who will work 
directly with ELT schools to provide guidance, leadership and 
decision-making support on challenges related to implementing 
ELT 

• District supports ELT schools in making instructional and 
curricular changes related to the ELT redesign 

• District helps institute additional time for teacher collaboration as 
part of the ELT redesign 

• District supports school’s efforts to provide professional 
development and teacher collaboration that are aligned with ELT 
redesign 

• District provides support and leadership around specific logistical 
issues (staffing, school schedule, sharing of staff, negotiated 
agreements, transportation, food service, etc.) that may need group 
problem solving to address 

• The district aligns resources for ELT (support staff, technology, 
time, and dollars) to support the school's instructional focus and 
the implementation of the ELT redesign 


• Superintendent is seen as an active supporter of the ELT work being a catalyst for 
school and district-wide improvement 

• Expansion of the ELT work has been considered as an element in the development of 
the long-term strategic plan of the district 

• Lessons learned by schools implementing an ELT redesign are intentionally shared 
with all schools and replicated when possible. 

• District communicates to the wider community how the implementation of ELT is 
linked to growth in student learning and progress on district and school-wide 
achievement goals 


Focus on 

Results 


♦ 5901 Warner Ave #441, Huntington Beach, CA 92649 

♦ 198 Tremont Street, #408, Boston, MA 02116 

♦ Phone: 888-743-1076 ♦ Fax: 508-869-0562 


Page 10 


Finding resuits-oriented solutions to the challenges 
of educating all students at high levels. 

www.focusonresults.net 


pg. 170 | Appendix A. Exhibits for Chapter 1 


Abt Associates Inc. 



ELT Evaluation Year 5 Integrated Report 


Appendix B. Exhibits and Model Specifications for Chapter 2 


Exhibit B.1 : Characteristics of Sample Schools, by Cohort and ELT Status 


Number of Schools 



Cohort 1 

Cohort 2 

Cohort 3 


ELT 

MC 

ELT 

MC 

ELT 

MC 


Grade Span 


Elementary school 

1 

1 

3 

3 

2 

2 

K-8 school 

3 

4 

1 

1 

0 

1 

Middle school 

4 

3 

1 

1 

3 

3 


School Location 


City 

7 

7 

1 

2 

1 

2 

Suburb 

1 

1 

1 

2 

4 

4 

Town 

0 

0 

2 

1 

0 

0 

Rural 

0 

0 

1 

0 

0 

0 


School Size 


600 students or more 

3 

1 

2 

3 

2 

3 

400-599 students 

3 

5 

2 

1 

3 

3 

200-399 students 

2 

2 

0 

1 

0 

0 

Fewer than 200 students 

0 

0 

1 

0 

0 

0 


Low Income Student Population 


75 percent or more 

4 

4 

1 

2 

4 

2 

50-74 percent 

4 

2 

4 

3 

1 

4 

Less than 50 percent 

0 

2 

0 

0 

0 

0 


Minority Student Population 


75 percent or more 

6 

3 

0 

0 

1 

1 

50-74 percent 

1 

4 

2 

3 

2 

2 

25-49 percent 

1 

1 

1 

1 

2 

3 

Less than 25 percent 

0 

0 

2 

1 

0 

0 


SPED Student Population 


20 percent or more 

4 

3 

2 

2 

1 

6 

10-19 percent 

4 

5 

3 

3 

3 

0 

Less than 1 0 percent 

0 

0 

0 

0 

1 

0 


LEP Student Population 


20 percent or more 

5 

3 

1 

1 

1 

0 

10-19 percent 

0 

2 

1 

2 

2 

2 

Less than 1 0 percent 

3 

3 

3 

2 

2 

4 


Met Aggregate Adequate Yearly Progress (AYP) in 2011 


English language arts 

2 

0 

3 

1 

2 

3 

Math 

1 

1 

1 

2 

0 

3 


ELA Accountability Status in 2011 


No status (AYP met for previous two years) 

1 

2 

1 

1 

0 

1 

Identified for improvement, corrective action, or 
restructuring 

7 

6 

4 

4 

5 

5 


Math Accountability Status in 2011 


No status (AYP met for previous two years) 

0 

1 

1 

2 

0 

1 

Identified for improvement, corrective action, or 
restructuring 

8 

7 

4 

3 

5 

5 


EXHIBIT READS: There is one ELT elementary school and one matched comparison elementary school in the study sample 
in Cohort 1 . 

Source: MA ESE website (http://profiIes.doe.mass.edu/). 

Sample: Full sample of 37 schools, including 18 ELT and 19 matched comparison schools. 


Abt Associates Inc. 


Appendix B. Exhibits and Model Specifications for Chapter 2 | pg. 171 




ELT Evaluation Year 5 Integrated Report 


B.2: Technical Appendix with Detailed Model Specifications 

Additional Details on the Data Used for Choosing Matched Comparison Schools 

Data for matching were downloaded from publicly available datasets on the ESE website. To identify 
potential matched comparison schools, data from the year immediately prior to ELT implementation were 
used. For Cohort 1 schools, 2005-06 data were referenced, for Cohort 2 schools, 2006-07 data were 
referenced, and for Cohort 3 schools, 2007-08 data were referenced. 

As described in Chapter 2, ELT schools were matched to non-ELT schools within district and grade span 
based on prioritization of selected matching variables including but not limited to the Composite 
Performance Index (CPI) and Adequate Yearly Progress (AYP). The CPI is a “100-point index combining 
the scores of students who take standard MCAS tests (the Proficiency Index) with the scores of those who 
take the MCAS-Alternate Assessment (MCAS-Alt) (the MCAS-Alt Index) and is a measure of the extent 
to which students are progressing toward proficiency in ELA and mathematics, respectively.” 
(http://www.doe.mass.edu/sda/ayp/2008/glossary.doc) 

“The federal No Child Left Behind Act (NCLB) requires all schools and districts to meet or exceed 
specific student performance standards in English language arts (ELA) and mathematics by the year 2014. 
AYP determinations are issued yearly based on the performance of all students (the “aggregate”) and for 
individual student groups (“subgroups”) to gauge the interim progress toward the attainment of those 
goals. To make AYP, districts and schools must meet a student participation requirement, an additional 
attendance or graduation requirement, and either the State's annual performance target for that subject or 
the district, school or group's own annual improvement target.” 
(http://www.doe.mass.edu/sda/ayp/2008/glossary.doc) 

Districts, schools, and student subgroups are expected to make AYP in ELA and mathematics. Districts, 
schools, or subgroups that make AYP in consecutive years have no NCLB Accountability Status. Those 
that do not make AYP for two consecutive years or more may be identified for Improvement, Corrective 
Action, or Restructuring for students in the aggregate or for one or more student subgroups. Improvement, 
Corrective Action, and Restructuring statuses trigger specific consequences 
(http://www.doe.mass.edu/sda/ayp/2008/glossary.doc). 

Model Specifications 

The following statistical models were fit to produce findings addressing the research questions posed in 
this report. 75 

Models used to Estimate the Effect of ELT on Non- Academic Outcomes 

Model estimating differences in time use between ELT and matched comparison schools 

Data: Interview data from ELT and matched comparison principals conducted in Spring, 2011. 


Yj* = /o+ Yi ELT jk + Pk + £ jk 


75 Because year five coefficients rely solely on data from the first cohort of study schools, the main text describes 
results only through year four and results of models that include terms to estimate year five preliminary results 
are presented in Appendices D and E. 
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Where: 

Yj k - outcome measure for school j in matched pair k in spring 2011 

ELTj k - one if the school is in an ELT school, and 0 otherwise 

p k - matched-pair dummy variables, modeled by a series of indicator variables for each 

pair of matched schools, k 
£j k - a school specific error term 


The difference in time is directly estimated as y x , the parameter estimate on the ELT term. 


Combined Model estimating the effect of ELT on teacher survey responses 

Data: teacher-level responses from ELT and matched comparison schools to survey items in the 2010-11 
school year. 

M N 

Yijk = To + 7 1 ELT j k + y Um Teach _ de in ijk + I Yu M+n Sch _ char ” + p k + p jk + £ ijk 

m = 1 n = 1 


Where: 

Y ijk 

ELT jk 

Teach _ dem"‘ k 
Sch _ char'j k 
Pk 

Pjk 


- outcome measure for teacher i in school j in matched pair k in spring 2011 
= one if the teacher is in an ELT school, and 0 otherwise 

= m th teacher-level demographic characteristic 
= n lh school-level demographic characteristic 

= matched-pair dummy variables, modeled by a series of indicator variables for each 
pair of matched schools, k 

= a school-level error term that accounts for the fact that teachers are clustered within 
schools and pairs 
= a teacher-level error term 


The effect of ELT is directly estimated as y x , the parameter estimate on the ELT term. 

Combined Model estimating the effect of ELT on student survey responses 

Data: student-level responses from ELT and matched comparison schools to survey items in the 2010-11 
school year. 


M N 

Ym = y 0 + y x ELT jk + £ y Um Stud _ dem'” k + £ y i+M Jch _ char ; + p k + p jk + v tjk + s itjk 

m = 1 n = 1 


Where: 

Y it j k - outcome measure for student i in classroom of teacher t in school j in matched pair 

k in spring 2011 

ELTj k - one if the student is in an ELT school, and 0 otherwise 
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Stud _ dem'" k 

- m th student-level demographic characteristic 

Sch _ char'- k 

= n th school-level demographic characteristic 

Pk 

= matched-pair school fixed effects, modeled by a series of indicator variables for 
each pair of matched schools, k 

Pjk 

= a school-level error term that accounts for the fact that students are clustered within 
schools 


= a teacher-level error term that accounts for the fact that students are clustered 
within classrooms of a particular teacher 

£ itjk 

= a student-level error term 


The effect of ELT is directly estimated as y l , the parameter estimate on the ELT term. 

Model estimating the effect of ELT on student behavioral indicators 

Data: Longitudinal, student-level behavioral measures across grades in years prior to and after ELT. 

The model includes year fixed effects and student fixed effects, and indicator variables that designate 
whether the school was an ELT school in a post year, as specified below. 



Y y =y l (ELT_Pn iy )+y 2 (ELT_PY2 iy )+y 3 (ELT_PY3 iy )+ 
Y a {ELT_ P YA iy )+y 5 (ELT_ PY5 iy )+S i +T y +s iy 

Where: 

Yiy 

ELT_PYl iy 

- outcome measure for student i in year y 

- one if the student measure is from an ELT school in its first post year after the 
implementation of ELT, and 0 otherwise 

ELT_PY2 iy 

= one if the student measure is from an ELT school in its second post year after the 
implementation of ELT, and 0 otherwise 

ELT_PY3 iy 

= one if the student measure is from an ELT school in its third post year after the 
implementation of ELT, and 0 otherwise 

ELT_PY4 iy 

= one if the student measure is from an ELT school in its fourth post year after the 
implementation of ELT, and 0 otherwise 

ELT_PY5 iy 

= one if the student measure is from an ELT school in its fifth post year after the 
implementation of ELT, and 0 otherwise 

Si 

Ty 

£ iy 

= student fixed effect, modeled by a series of indicator variables for each student 
= year fixed effect, modeled by a series of indicator variables for each year 
= a student-level error term 


The effect of ELT after one year of implementation is directly estimated as y x , the parameter estimate on 
the ELT_PY 1 term; the effect after two years of implementation is estimated as y 2 , the parameter estimate 
on the ELT_PY2 term; the effect after three years of implementation is estimated as y 3 , the parameter 
estimate on the ELT_PY3 term; the effect after four years of implementation is estimated as y 4 , the 
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parameter estimate on the ELT_PYA term; and the effect after five years of implementation is estimated as 
y 5 , the parameter estimate on the ELT_PY5 term. Standard errors are corrected for the correlations 

between student outcomes within schools (within and across years) beyond what is captured by the year 
fixed effects using the cluster-robust variance estimator (also known as the “sandwich” standard errors; 
White, 1984 and Liang and Zeger, 1986). 

Models used to Estimate the Effect of ELT on School Characteristics and Academic Outcomes 
Model estimating the effect of ELT on characteristics of students and teachers in schools 

Data: Longitudinal, school-level data on the number or proportion of students and teachers with various 
characteristics in years prior to and after ELT. 

The model includes school fixed effects, year fixed effects, and indicator variables that designate whether 
a school was an ELT school in a post year, as specified below. 

^ y = r fELT_P}l jy )+r 2 (ELT_P^ y )+r 3 (ELT_P B^+ 
r 4 (ELT_PY4.)+r 5 (ELT_P}5 j ^+V J +T y + £jy 

Where: 

Yjy 

ELT_PYlj y 
ELT_PY2 jy 
ELT_PY3 jy 
ELT_PY4 jy 
ELT_PY5 jy 

Vj 

Ty 


= outcome measure for school j in year y 

— one if the school j is an ELT school in its first post year after the implementation of 
ELT, and 0 otherwise 

= one if the school is an ELT school in its second post year after the implementation 
of ELT, and 0 otherwise 

= one if the school is an ELT school in its third post year after the implementation of 
ELT, and 0 otherwise 

= one if the school is an ELT school in its fourth post year after the implementation 
of ELT, and 0 otherwise 

= one if the school is an ELT school in its fifth post year after the implementation of 
ELT, and 0 otherwise 

= school fixed effects, modeled by a series of indicator variables for each school 

= year fixed effect, modeled by a series of indicator variables for each year 

= the usual school-year specific error term 


The effect of ELT after one year of implementation is directly estimated as y x , the parameter estimate on 
the ELT_PY 1 term; the effect after two years of implementation is estimated as y 2 , the parameter estimate 
on the ELT_PY2 term; the effect after three years of implementation is estimated as / 3 , the parameter 

estimate on the ELT_PY3 term; the effect after four years of implementation is estimated as y 4 , the 
parameter estimate on the ELT_PYA term; and the effect after five years of implementation is estimated as 
y 5 , the parameter estimate on the ELT_PY5 term. Standard errors are corrected for the correlations 

between student outcomes within schools (within and across years) beyond what is captured by the school 
and year fixed effects using the cluster-robust variance estimator (also known as the “sandwich” standard 
errors; White, 1984 and Liang and Zeger, 1986). 
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Estimating differences between ELT and matched comparison schools’ student mobility 

Data: Longitudinal, student-level data across grades in years prior to and after ELT. 


*1 ~ * 2 


Where: 

x 1 — x 2 - difference in means between ELT and matched comparison schools 

s x 1 -x 2 - the standard error 

Given that student mobility is presented in the main text as an alternative metric for assessing the 
relationship between ELT and school outcomes, student mobility analyses are exploratory and test for the 
difference in means by implementation year is using the probability (p-value) associated with the t- 
statistic. 

Model estimating the effect of ELT on student MCAS scores 

Data: Longitudinal, student-level test scores by grade in years prior to and after ELT. 

The model includes school fixed effects, year fixed effects, student level demographic characteristics, and 
indicator variables that designate whether a school was an ELT school in a post year, as specified below. 
The error term also adjusts for the correlation of student scores within a school in a school year. 

Yuy = n(ELT_PYl ijy ) + y 2 (ELT_PY2 ijy ) + h (ELT _PY3 ijy ) + y 4 (ELT _PY4 ijy ) + 

r 5 (ELT_PY5 ijy ) + f^ r6+k (Stud_derrt jy ) + V j + T y + £ijy 

k = 1 


Where: 

Y 

1 >jy 

ELT_PYl ijy 


ELT_PY2, jy 


ELT_PY3 ijy 


ELT_PY4 ijy 


ELT_PY5 ijy 


Stud _ dem k ijy 

Yj 

Ty 


= outcome measure for student i in school j in year y 

— one if the student measure is from an ELT school in its first post year after the 
implementation of ELT, and 0 otherwise 

= one if the student measure is from an ELT school in its second post year after the 
implementation of ELT, and 0 otherwise 
= one if the student measure is from an ELT school in its third post year after the 
implementation of ELT, and 0 otherwise 

= one if the student measure is from an ELT school in its fourth post year after the 
implementation of ELT, and 0 otherwise 
= one if the student measure is from an ELT school in its fifth post year after the 
implementation of ELT, and 0 otherwise 
= k lh student-level demographic characteristic 

= school fixed effects, modeled by a series of indicator variables for each school 
= year fixed effect, modeled by a series of indicator variables for each year 
= the usual student-level error term 
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The effect of ELT after one year of implementation is directly estimated as / l , the parameter estimate on 
the ELT_PY 1 term; the effect after two years of implementation is estimated as y 2 , the parameter estimate 
on the ELT_PY2 term; the effect after three years of implementation is estimated as / 3 , the parameter 

estimate on the ELT_PY3 term; the effect after four years of implementation is estimated as y A , the 
parameter estimate on the ELT_PYA term; and the effect after five years of implementation is estimated as 
y 5 , the parameter estimate on the ELT_PY5 term. Standard errors are corrected for the correlations 

between student outcomes within schools (within and across years) beyond what is captured by the school 
and year fixed effects using the cluster-robust variance estimator (also known as the “sandwich” standard 
errors; White, 1984 and Liang and Zeger, 1986). 
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Exhibit C.1: Implementation Index Summary Matrix 


Year 5 
School 
Number 

ELT 

Raw 

Implementation 

Index 

1. 

School-wide 

Academic 

Focus 

2. 

Core 

Academics 

2. 

Core 

Academics\ln 
addition to time 

3. 

Enrichment 

Activities 

3. 

Enrichment 
ActivitiesMn 
addition to time 

4. 

Teacher 
leadership and 
collaboration 

4. 

Teacher 
leadership and 
collaborationMn 
addition to time 

5. 

School 

Leadership 

6. 

ELT 

Support 

# of 
0's 

# of 
1's 

#of 

2's 

# of 
3's 

j 

1 

22 

3 

3 

3 

3 

2 

2 

3 

3 

1 

0 

0 

2 

6 

0 

1 

19 

3 

3 

2 

3 

3 

2 

2 

1 

2 

0 

i 

3 

4 

1 

1 

15 

0 

3 

1 

3 

2 

3 

1 

2 

1 

1 

2 

2 

3 

A 

1 

15 

3 

3 

3 

1 

2 

1 

1 

1 

3 

0 

4 

1 

3 

B 

1 

14 

0 

2 

2 

2 

2 

2 

2 

2 

2 

1 

0 

7 

0 

H 

1 

13 

2 

1 

1 

3 

2 

3 

1 

0 

0 

1 

3 

2 

2 

C 

1 

12 

0 

2 

0 

1 

3 

2 

2 

2 

2 

2 

1 

4 

1 

E 

1 

11 

0 

1 

1 

3 

3 

0 

1 

2 

3 

2 

3 

1 

2 

D 

1 

11 

0 

1 

2 

3 

1 

0 

2 

2 

3 

2 

2 

3 

1 

K 

1 

10 

0 

1 

1 

3 

2 

1 

1 

1 

3 

1 

5 

1 

1 

Q 

1 

9 

0 

2 

0 

1 

2 

1 

1 

2 

2 

2 

3 

3 

0 

R 

1 

9 

0 

0 

2 

1 

2 

0 

1 

3 

3 

3 

2 

2 

1 

M 

1 

8 

0 

0 

1 

3 

0 

1 

1 

2 

1 

3 

3 

1 

1 

L 

1 

8 

0 

1 

1 

3 

1 

1 

0 

1 

3 

2 

5 

0 

1 

G 

1 

7 

0 

2 

0 

0 

1 

0 

1 


3 

4 

2 

1 

1 

F 

1 

6 

0 

2 

0 

0 

1 

0 

1 

2 

3 

4 

2 

2 

0 

N 

1 

5 

0 

1 

0 

0 

2 

0 

1 

1 

2 

4 

3 

1 

0 

P 

1 


3 


0 


1 

0 

2 

1 

1 

2 

2 

1 

1 

S 

0 

12 

3 

0 

1 

2 

3 

1 

1 

1 


1 

4 

1 

2 

T 

0 

10 

2 

1 

0 

3 

2 

0 

1 

1 


2 

3 

2 

1 

U 

0 

10 

3 

1 

0 

1 

0 

1 

1 

3 


2 

4 

0 

2 

V 

0 

10 

3 

0 

1 

0 

1 

1 

1 

3 


2 

4 

0 

2 

w 

0 

9 

3 

2 

0 

0 

0 

0 

2 

2 


4 

0 

3 

1 

X 

0 

8 

0 

0 

2 

0 

3 

0 

0 

3 


5 

0 

1 

2 

Y 

0 

8 

0 

3 

0 

0 

1 

0 

1 

3 


4 

2 

0 

2 

z 

0 

7 

0 

3 

0 

0 

0 

2 

1 

1 


4 

2 

1 

1 

AA 

0 

7 

0 

0 

2 

0 

0 

2 

0 

3 


5 

0 

2 

1 

BB 

0 

6 

0 

1 

0 

0 

1 

2 

1 

1 


3 

4 

1 

0 

CC 

0 

6 

0 

1 

0 

0 

1 

0 

1 

3 


4 

3 

0 

1 

DD 

0 

5 

0 

2 

0 

0 

1 

0 

1 

1 


4 

3 

1 

0 

EE 

0 

4 

0 

1 

0 

0 

0 

0 

1 

2 


5 

2 

1 

0 

FF 

0 

3 

0 

0 

2 

0 

0 

0 

0 

1 


6 

1 

1 

0 

GG 

0 

3 

0 

1 

0 

0 

0 

0 

1 

1 


5 

3 

0 

0 

HH 

0 

2 

0 

0 

0 

0 

0 

0 

1 

1 


6 

2 

0 

0 

II 

0 



0 


0 






2 

0 

0 

0 

JJ 

0 



3 


2 






0 

0 

1 

1 

KK 

0 


0 

1 ■ 

0 


0 

0 

1 

3 


4 

1 

0 

1 


OVERALL SUM: | 28 | 47 | 28 | 41 | 45 | 28 | 39 | 64 | 38 


ELT Criterion Average 

0.78 

1.65 

1.11 

1.94 

1.78 

1.06 

1.33 

1.72 

2.11 

MC Criterion Average 

0.88 

1.06 

0.47 

0.44 

0.76 

0.53 

0.88 

1.94 

N/A 


Overall Average Raw Score | 8.67 


Average ELT Raw Score | 1 1.44 


Average MC Raw Score | 6.88 
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Criterion 1 

Criterion 2 
Time 

Criterion 2 
AT 

Criterion 3 
Time 

Criterion 3 
AT 

Criterion 4 
Time 

Criterion 4 
AT 

Criterion 5 

Criterion 6 

Overall 

# of 0's 

25 

9 

18 

17 

10 

18 

4 

i 

1 

# of 1 's 

0 

12 

8 

5 

10 

8 

24 

14 

4 

# of 2's 

2 

7 

7 

3 

10 

7 

6 

10 

5 

# of 3's 

8 

7 

2 

10 

5 

2 

1 

10 

8 

ELT 

# of 0's 

13 

2 

6 

3 

1 

7 

1 

1 

1 

# of 1 's 

0 

6 

7 

4 

5 

6 

12 

7 

3 

# Of 2'S 

2 

5 

3 

2 

9 

2 

4 

8 

4 

# of 3's 

4 

1 

1 

7 

3 

1 

0 

3 

8 

MC 

# of 0's 

12 

7 

12 

14 

9 

11 

3 

0 

0 

# of 1 's 

0 

6 

1 

1 

5 

2 

12 

7 

0 

# of 2'S 

0 

2 

3 

1 

0 

3 

1 

2 

0 

# of 3's 

2 

3 

0 

0 

1 

0 

0 

6 

0 
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Information regarding differences between ELT cohorts is presented below. It should be noted, as shown 
in Exhibit D.l, that there it is difficult to interpret any cohort differences given the small (and different) 
number of schools in each cohort as well as the distribution of grades across cohorts. 


Exhibit D.l: Number of ELT Schools by Cohort, 2010-11 



Total Number of 

Total Number of 

Total Number of 


Schools 

Elementary Schools 

Middle Schools 

ELT Cohort 

(N=18) 

(N=10) 

(N=8) 

1 

8 

4 

4 

2 

5 

4 

1 

3 

5 

2 

3 


Total Number of 

Total Number of 

Total Number of 


5 th & 8 th Grades 

5 th Grades 

8 th Grades 

ELT Cohort 

(N=23) 

(N=12) 

(N=11) 

1 

11 

4 

7 

2 

5 

4 

1 

3 

7 

4 

3 


Time Use in ELT Schools 

Overall Length of the ELT School Day and Time Allocated to Different Instructional Activities 

The school day was approximately 15 minutes longer, on average, for schools in Cohorts 1 and 3 than 
Cohort 2 schools (not shown). On average, Cohort 2 schools allocated about 20 minutes more to core 
academics each day (5 hours) than Cohort 1 schools (4 hours and 39 minutes), and about 12 minutes more 
than Cohort 3 schools (4 hours and 48 minutes) (not shown). Among the schools that offered enrichment 
classes, there was no difference between cohorts in amount of time allocated to enrichment. Among the 
schools that offered academic support classes, Cohort 1 allocated 1 1 more minutes to academic support 
each day (41 minutes) than Cohort 2 (30 minutes) and 8 more minutes than Cohort 3 (33 minutes). These 
daily differences in average amount of time spent in academic support extrapolate into 40-55 min more 
academic support each week. 

As shown in Exhibit D.2, teacher satisfaction with the amount of instructional time available in each of 
the core subject areas was general high (70 percent or more teachers reporting satisfaction) and there were 
no differences among cohorts. With regards to the amount of time available for physical activity during 
the day, however, teacher satisfaction was low, with less than half (41 percent) of teachers reporting 
satisfaction; among the three cohorts, Cohort 3 teachers were least satisfied with the amount of time 
during the day for physical activity (35 percent Cohort 3, 41 percent Cohort 2, and 47 percent Cohort 1 
reporting satisfaction). 
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Exhibit D.2: Teacher Satisfaction with Instructional Time Available for Core Academics and 


Physical Activity, by Cohort, 2010-11 

Percent of Teachers 


Overall 

Cohort 1 

Cohort 2 

Cohort 3 

Time available for instruction in ELA. 

90% 

92% 

91% 

85% 

Time available for instruction in math. 

93 

93 

95 

89 

Time available for instruction in science. 

81 

86 

72 

81 

Time available for instruction in social studies. 

71 

78 

65 

66 

Time available for physical activity. 

42 

47 

41 

35 


School-Wide Academic Focus 

According to principals, 16 of 18 ELT schools had established a school-wide academic focus. 76 Student 
reports of the school-wide academic focus are summarized in Exhibit D.3. In contrast to teacher and 
principal reports, the most common focus reported by students was math (31 percent), as was the case in 
2009- 10. 77 Twenty-one percent of students reported that there was no “special topic that every student is 
supposed to learn.” According to 1 1 percent of students, the focus was test taking skills. Compared to 
principals and teachers, far fewer students reported that the focus was reading or writing (6 and 5 percent, 
respectively). At the school level, student reports were inconsistent with teacher and principal reports at 
all but one school. Within most schools (14), there was not consistency among students. 


Exhibit D.3: Student Reports of Academic Focus, By School Configuration, 2010-11 



Percent of Students Reporting 

Overall 

Elementary 

Middle 

Math 

31% 

29% 

32% 

Test Taking Skills 

11 

8 

14 

Reading 

6 

8 

6 

Thinking Skills 

5 

4 

6 

Science 

5 

4 

6 

Writing 

5 

6 

5 

Vocabulary 

4 

1 

5 

Social Studies 

2 

2 

2 

There is no special topic 

21 

26 

17 

Other topic 

11 

14 

8 


EXHIBIT READS: In spring 2011, across all ELT schools, 31 percent of students reported that math was the “one special 
topic every student is supposed to learn this year.” 

Source: Abt Associates’ Surveys ofMA ELT School Students , Spring 2011, Item 7. 

Sample: 2,300 students from ELT schools. Nonresponse rate was 5.6 percent. 


76 Given the small number of ELT schools overall and by cohort, the discussion presents information in terms of 
number of schools rather than percentages. 

77 Students were asked on surveys to report on “the one special topic every student in your school is supposed to 
learn this year.” Students could choose one of eight topics listed, indicate there was no special topic, or write in 
another topic if their response was not listed. 
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All Cohort 3 schools had a focus. The majority of teachers (81 percent) reported that their school had an 
academic focus, with no substantial differences across cohorts. As reported by teachers, school academic 
focus areas varied somewhat by cohort (see Exhibit D.4). Cohort 3 teachers reported literacy more often 
than Cohorts 1 and 2 (46, 31 and 35 percent, respectively). 


Exhibit D.4: Teacher Reports of Academic Focus, By Cohorts, 2010-11 


Percent of Teachers Reporting 



Overall 

Cohort 1 

Cohort 2 

Cohort 3 

Literacy 

36% 

31% 

35% 

46% 

Math 

14 

17 

13 

10 

Higher Order Thinking Skills 

11 

10 

6 

19 

Vocabulary 

3 

3 

0 

5 

Science 

0 

- 

1 

- 

Test Taking Skills 

2 

3 

3 

1 

Writing 

9 

13 

8 

3 

Social Studies 

0 

- 

- 

1 

Other 

24 

22 

34 

16 

Literacy 

36 

31 

35 

46 


EXHIBIT READS: In spring 2011, across all ELT schools, 36 percent of teachers reported that literacy described “the 
school’s primary academic focus this year” 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 7. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 5.6 percent. 


According to principals, the academic focus was posted publicly at most schools (13 of 16). Principal 
interviews and site visits revealed that the academic focus was most frequently publicly available at 
Cohort 3 schools and least available at Cohort 1 schools. For example, all Cohort 3 schools posted the 
focus in the hallways, and three-quarters posted it in classrooms as well. Cohort 1 schools were most 
likely to report that the focus was not posted anywhere (3 schools compared to 0 schools in the other two 
cohorts). 


Enrichment 

Time 

According to principals, time for enrichment at ELT schools during 2010-11 ranged from 0 to 340 
minutes (or 5 hours and 40 minutes) per week, and schools allocated approximately 2 hours and 20 
minutes each week (139 minutes) for a typical student, on average. Cohort 1 allocated 181 minutes on 
average (approximately 3 hours), nearly an hour more than schools in Cohorts 2 and 3 (which allocated 
138 minutes and 120 minutes, respectively). 


Collaborative Planning Time 

There were modest differences in the frequency of participation in Collaborative Planning Time across 
Cohorts. As Exhibit D.5 displays, more cohort 2 teachers reported participating in collaborative planning 
time more frequently than once a week, than did teachers in cohorts 1 or 3. 
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Exhibit D.5: Frequency of Teacher Participation in Collaborative Planning Time, By Cohort, 
2010-11 


Overall 


Cohort 1 


Cohort 2 


Cohort 3 


16 

37 

28 

5 


13 

40 

25 

3 


23 

27 

30 

5 II 


12 

41 

31 

7 


0 20 40 60 80 100 

% Agree 

□ More than once a week BOnce a week □Twiceamonth nMonthly nNever 


EXHIBIT READS: In spring 2011, across all ELT schools, 16 percent of teachers reported that they participated in 
collaborative planning time more than once a week. 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Item 23. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 3.7 percent. 


Teacher satisfaction with the amount of time available for collaborative planning was fairly consistent 
across cohorts and school configurations, with about 60 percent of teachers indicating that they were 
satisfied. 


Professional Development 

Time and Frequency 

Across cohorts and school configurations, about 85 percent of teachers participated in at least 10 hours of 
professional development over the course of the year. Exhibit D.6 shows the distribution of teachers’ 
reported hours of professional development participation by cohort. Overall, half of teachers surveyed 
reported having completed more than 26 hours of professional development over the course of the school 
year. There were some differences in levels of reported participation across cohorts; 40 percent of Cohort 
2 teachers, compared to nearly 60 percent of cohort 1 teachers, reported having participated in 26 or more 
hours of professional development. A similar pattern was observed for teachers who had participated in 
50 hours or more of professional development, with considerably more teachers from Cohorts 1 than 
Cohort 2 teachers reporting participation (27 percent of Cohort 1 teachers and 17 percent of Cohort 2 
teachers). Nearly twice as many Cohort 2 teachers as Cohort 1 or 3 teachers reported having engaged in 
under 10 hours of professional development. 
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Exhibit D.6: Teacher Time Spent in Professional Development, by Cohort, 2010-11 


100 

80 

60 


40 


20 


JTu JTu Jin 


Overall 


Cohort 1 


Cohort 2 


□ Fewer than 10 hours 

□ 10-25 hours 

□ 26-50 hours 

□ 51-75 hours 
■ 76-100 hours 

□ More than 100 hours 



Cohort 3 


EXHIBIT READS: In spring 2011, across all ELT schools, 13 percent of teachers reported participating in fewer than 10 
hours of professional development over the course of the 2010-11 school year. 

Source: Abt Associates’ Sun’eys ofMA ELT School Teachers , Spring 2011, Item 29. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 5.0 percent. 


Across all three of the most common professional development activities, fewer cohort 2 teachers than 
cohort 1 or 3 teachers reported participating in professional development. See Exhibit D.7 below. 


Exhibit D.7: Sources Used by ELT Teachers to Support Instruction, by Cohort, 2010-11 



Percent Used 


Overall 

Cohort 1 

Cohort 2 

Cohort 3 

Out-of-school professional development 

53% 

58% 

45% 

56% 

Peer support (e.g., peer coaching, peer modeling, peer 
demonstration lessons) 

49 

55 

37 

53 

Academic coaches in school 

48 

51 

40 

52 

Principal and/or other administrative support 
(e.g., coaching, modeling) 

27 

29 

24 

30 

Special education aide 

27 

22 

33 

30 

Partner staff 

26 

23 

28 

28 

Technical assistance from outside district 

9 

12 

8 

5 

Other 

3 

3 

3 

3 


EXHIBIT READS: In spring 2011, across all ELT schools, 53 percent of teachers reported using out-of-school professional 
development to support instruction during the 2010-11 school year. 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 28. 

Sample: 918 teachers from ELT schools. The binary nature of the survey items does not allow for the reporting of 
nonresponse rates. 
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Teachers’ satisfaction with the amount of time for PD, however, varied across cohorts, with 55 percent of 
cohort 2 teachers reporting satisfaction, and 75 percent of cohort 1 teachers reporting satisfaction. Cohort 
3 fell between these two, with 64 percent of teachers satisfied. 


Exhibit D.8: Teacher Satisfaction with Professional Development, by Cohort, 2010-11 


Satisfaction with amount of time for PD 


In-school professional development 
activities contribute to the development 
of a professional learning community of 
teachers. 



% Agree 


EXHIBIT READS: In spring 2011, across all ELT schools, 66 percent of teachers reported being satisfied with the amount 
of time for professional development in their school. 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 32c and 27c. 

Sample: 918 teachers from ELT schools. Nonresponse rates across items ranged from 9.2 to 24.8 percent. 
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Leadership 

Teacher Perceptions of District Leadership 

There were some modest differences across cohorts and school types; as Exhibit D.9 shows, Cohort 3 and 
elementary teachers’ responses were more positive, on average, about district leadership than others. 


Exhibit D.9: Teacher Attitudes about District Leadership, by School Type. 2010-11 



Percent Agree 


Overall 

Cohort 1 

Cohort 2 

Cohort 3 

The district leadership effectively manages our schools. 

62% 

60% 

56% 

70% 

The district leadership communicates a clear vision for our 
district's schools. 

73 

73 

65 

80 

The district leadership is interested in the professional 
development of teachers at this school. 

69 

72 

58 

76 

The district leadership provides timely guidance on 
instructional practice, curriculum, etc. 

60 

64 

51 

65 

The district leadership is responsive to this school and 
teacher concerns. 

51 

55 

40 

57 

The district leadership actively supports ELT 
implementation at this school. 

85 

85 

83 

89 


EXHIBIT READS: In spring 2011, across all ELT schools, 62 percent of teachers agreed that “The district leadership 
effectively manages our schools.” 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 7cc through 7hh. 

Sample: 918 teachers from ELT schools. Nonresponse rate ranged from 13.7 to 21.1 percent. 


Teacher Perceptions of Principal Leadership 

Over two-thirds of teachers generally agreed that their principals provided quality leadership, evidenced 
by communication, expectations, engagement with student progress, and receptiveness to feedback. Of 
note, fewer teachers in Cohort 3 indicated that their principals were effective managers who made the 
school run smoothly; that their principals were instructional leaders; and/or that their principals provided 
opportunities for staff leadership or recognition of staff success than did teachers in Cohorts 1 and 2 (see 
Exhibit D. 10). 
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Exhibit D.10: Teacher Perceptions of Principal Leadership, by School Type, 2010-11 


7p. The principal and/or school leadership 
team provide opportunities for staff 
leadership and/or recognition of staff 
successes. 


7k. The principal and/or school leadership 
team are instructional leaders in our school. 


7g. The principal and/or school leadership 
team are effective managers who make the 
school run smoothly. 


72 


69 

I 73 
1 73 


83 


0 20 40 60 80 100 



□ Overall 

□ Cohort 1 

□ Cohort 2 

□ Cohort 3 



% Agree 


EXHIBIT READS: In spring 2011, across all ELT schools, 74 percent of teachers agreed that “The principal and/or school 
leadership team were effective managers who make the school run smoothly.” 

Source: Abt Associates’ Surveys ofMA ELT School Teachers, Spring 2011, Item 7g, 7k, and 7p. 

Sample: 918 teachers from ELT schools. Nonresponse rates across items ranged from 4.0 to 7.6 percent. 


Homework Policies in ELT Schools 

Another important issue related to core academic instruction is homework, which can allow students to 
practice what they learn in their academic classes and further their skills. One potential challenge for ELT 
schools is balancing the value of practice or learning via homework with the reduced amount of time 
students have to complete homework once they get home, coupled with the possibility that students may 
be more fatigued when they leave school. On the other hand, a potential benefit of students being at 
school for longer each day is that they may have the opportunity to complete homework before they leave 
school. 

Overall, about two-thirds of teacher survey respondents (65 percent) reported that their school had a 
homework policy such that homework was regularly assigned. About one-third of teachers reported that 
there was no school- wide homework policy or that the school-wide homework policy was that little or no 
homework is assigned. 

Student responses about homework policies appeared to be consistent with those of teachers. Just over 
two-thirds (7 1 percent) of student survey respondents reported that they had homework on the last day 
they were in school. 

Overall, the majority of students who had homework reported that they did their homework (83 percent). 
Of students who did their homework, as shown in Exhibit D. 1 1, overall, about 40 percent reported 
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spending less than 30 minutes and about 40 percent reported spending between 30 and 60 minutes doing 
homework at home. About 40 percent reported spending no time and about 40 percent reported spending 
less than 30 minutes doing homework in school. In general, relatively few students reported spending 
more than an hour on homework whether at home or in school. 


Exhibit D.11: Student Reports of Total Time Spent on Homework on a Typical Day, 2010-11 



□ No time D30 minutes or less □ Between 30 minutes and 1 hour DMore than 1 hour 


EXHIBIT READS: In spring 2011, across ELT schools, 40 percent of students reported that they spent a total of 30 minutes or 
less working on homework at home on a typical day. 

Source: Abt Associates’ Surveys of MA ELT School Students, Spring 2011, Item 6c. 

Sample: 2,300 students from ELT schools. Nonresponse rate was 54.3 percent. 
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More than two-thirds (70 percent) of teachers indicated that homework completion rates were a problem 
area at their school, and more middle than elementary school teachers reported homework completion 
was problematic (85 percent and 57 percent). 


Exhibit D.12: Teacher Perceptions of Students’ Homework Completion Rates as a Problem 
Area, by School Configuration, 2010-11 






Overall 


70 

Elementary 


57 


Middle 


85 






0 20 40 60 80 100 


% Agree or Strongly Agree 


EXHIBIT READS: In spring 2011, across all ELT schools, 70 percent of teachers reported that they agreed or strongly 
agreed that homework completion rates were problematic. 

Source: Abt Associates’ Sun’eys ofMA ELT School Teachers, Spring 2011, Item 6c. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 19.7 percent. 
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As shown in Exhibit D.13, more Cohort 1 and 2 teachers reported regular homework assignment than 
Cohort 3 teachers. 


Exhibit D.13: Homework Policy in ELT Schools, by Cohort, 2010-11 



% Teacher Response That Homework is Regularly Assigned 


EXHIBIT READS: In spring 2011, across all ELT schools, 65 percent of teachers reported that they have a homework 
policy and that the homework policy is that homework is regularly assigned. 

Source: Abt Associates’ Sun’eys ofMA ELT School Teachers, Spring 2011, Item 11. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 5.5 percent. 


Student responses about homework policies appeared to be consistent with those of teachers. The 
majority of Cohort 1 and 2 students reported having homework, and just slightly more than half of Cohort 
3 students reported having homework (78, 81, and 57 percent, respectively, not shown). 

Among the three cohorts, more Cohort 2 than Cohort 1 and 3 students reported that they did their 
homework (91, 80 and 83 percent, respectively, not shown). 

As shown in Exhibit D.14, in general, relatively few students reported spending more than an hour on 
homework whether at home or in school; the one notable difference between cohorts is that more Cohort 
1 than Cohort 2 and 3 students reported spending more than an hour on homework at home (24, 13, and 
14 percent, respectively). 
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Exhibit D.14: Student Reports of Total Time Spent on Homework on a Typical Day, by Cohort, 
2010-11 


At Home 



□ No time B30 minutes or less □ Between 30 minutes and 1 hour □ More than 1 hour 


In School 



EXHIBIT READS: In spring 2011, across ELT schools, 40 percent of students reported that they spent a total of 30 minutes or 
less working on homework at home on a typical day. 

Source: Abt Associates’ Sun’eys ofMA ELT School Students, Spring 2011, Item 6c. 

Sample: 2,300 students from ELT schools. Nonresponse rate was 54.3 percent. 
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As shown in Exhibit D.15, more Cohort 3 than Cohorts 1 and 2 teachers (80 percent, 69 percent, and 63 
percent, respectively) agreed that homework completion rates were problematic. 


Exhibit D.1 5: Teacher Agreement on Whether Homework Completion Rates Were Problematic, 
by Cohort, 2010-11 



% Agree or Strongly Agree 

EXHIBIT READS: In spring 2011, across all ELT schools, 70 percent of teachers reported that they agreed or strongly 
agreed that homework completion rates were problematic. 

Source: Abt Associates’ Sun’eys ofMA ELT School Teachers, Spring 2011, Item 6c. 

Sample: 918 teachers from ELT schools. Nonresponse rate was 19.7 percent. 
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Appendix E. Exhibits for Chapter 5 


Exhibit E.1: Participation in Non-instructional Activities, 2010-11 



Percent of 
Respondents 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Actual 
ELT Mean 

Non-ELT 

Mean 

Participation in Structured Extracurricular Activities 




Attending an academic club (science club, math club, etc.) 

Participating in an Honor Society 

Participating in student government 

Attending a non-academic club (chess club, cheerleading, 
computer club) 

Playing on a sports team, practicing a sport, or exercising 

Participating in arts, music, theater, or dance 
lessons/activities 

Working on a school newspaper or magazine 
Participation in After-School Activities 

Do you go to any after-school activities this school year? 

How often do you usually go to any after-school activity? 

How much time do you usually spend at after-school 
activities each day you attend? 

Participation in Recreational or Leisure Activities 

Playing outside for fun 

Spending time online using a computer 

Texting or talking to friends 

Watching TV or playing video games 

Spending time with friends 

Time Spent At Home With Friends And Family 


Home alone or with younger siblings 

43 

39 

3.6 

0.316 

With your friends and no adults 

47 

49 

-1.4 

0.701 

With an older sibling or parent 

70 

69 

0.7 

0.818 


EXHIBIT READS: Twelve percent of students in ELT schools indicated that they participated in an academic club, compared 
to an estimated 17 percent in the absence of ELT. The difference between the two groups was statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys ofMA ELT and Matched Comparison School Students, Spring 2011, Items 12n, 121, 12g, 12o, 
12a, 12b, 12f 13, 13b, 13c, 12h, 12r, 12p, 12q, 12d, 15a, 15b, 15c. 

Sample: 4840 students. Nonresponse rates across ELT and matched comparison student survey items ranged from 2.6 to 21.1 
percent. 


82 

88 

-6.4 * 

0.014 

80 

78 

2.6 

0.520 

87 

85 

1.7 

0.521 

88 

89 

-0.5 

0.864 

90 

90 

-0.1 

0.972 


39 

50 

-11.7 * 

0.025 

53 

61 

-8.0 

0.204 

86 

79 

6.6 

0.055 


12% 

17% 

-5.7 * 

0.020 

10 

12 

-2.0 

0.517 

11 

13 

-1.9 

0.467 

23 

24 

-1.4 

0.653 

70 

71 

-0.6 

0.860 

57 

57 

-0.3 

0.950 

12 

11 

0.1 

0.940 
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Exhibit E.2: Teacher Perceptions of School and District Leadership 



Percent of 
Respondents 


Statistical 


ELT Mean 

Non-ELT 

Mean 

Estimated 

Difference 

Significance 

(p-value) 

Leadership 



The principal and/or school leadership team are effective 
managers who make the school run smoothly 

75% 

80% 

-4.4 

0.620 

The principal and/or school leadership team are 
instructional leaders in our school 

70 

72 

-2.4 

0.701 

Level of Adaptability 



The principal and/or school leadership team are actively 
engaged in school improvement initiatives 

90 

94 

-4.2 

0.343 

The principal and/or school leadership team adapt or 
refine school policies and/or practices as needed 

78 

82 

-3.9 

0.519 

The principal and/or school leadership team monitors 
progress of students 

90 

87 

3.3 

0.535 

Communication 



The principal and/or school leadership team are receptive 
to feedback on teacher morale/school culture 

68 

78 

-10.2 

0.229 

The principal and/or school leadership team provide 
feedback about teacher performance 

72 

80 

-8.1 

0.157 

The principal and/or school leadership team communicate 
effectively about improving instructional practices 

74 

76 

-1.6 

0.828 

Support for Staff Development 



The principal and/or school leadership team provide 
opportunities for staff leadership and/or recognition of 
staff successes 

67 

80 

-12.6 

0.136 

The principal and/or school leadership team encourages 
the development of a professional learning community 

85 

83 

2.3 

0.650 

The principal and/or school leadership team are 
supportive of staff growth and development 

86 

86 

0.3 

0.950 

Perceptions of District Leadership 



District leadership effectively manages our schools 

62 

56 

6.6 

0.396 

District leadership provides timely guidance on 
instructional practices 

61 

55 

6.0 

0.283 

District leadership communicates a clear vision for our 
district's schools 

73 

69 

3.9 

0.571 

District leadership is interested in professional 
development of teachers at this school 

70 

66 

3.3 

0.613 

District leadership is responsive to this school and teacher 
concerns 

52 

49 

2.6 

0.748 


EXHIBIT READS: Seventy-five percent of teachers in ELT schools reported that the principal and/or school leadership team 
were effective managers who made the school run smoothly, compared to an estimated 80 percent in the absence of ELT. The 
difference between the two groups was not statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Items 7g, 7k, 7h, 7i, 7n, 7j 7m, 71, 7p, 27e, 7o, 7cc, 
7ff, 7dd, 7ee, 7gg; Matched Comparison Teacher Surveys, Items 7e, 7i, 7f, 7g, 71, 7h, 7k, 7j, 7n, 26e, 7m, 7v, 7y, 7w, 7x, 7z. 
Sample: 1609 teachers. Nonresponse rates across ELT and matched comparison teacher survey items ranged from 5.8 to 20.7 
percent. 
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Exhibit E.3: Time Available for Instructional Activities 



Percent of 
Respondents 




Actual 
ELT Mean 

Estimated 

Non-ELT 

Mean 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Teacher Perceptions of Instructional Time 



Students have sufficient time to learn about subjects 

76% 

47% 

28.4 *** 

<.0001 

Length of day allows me time to accomplish teaching 
goals 

80 

66 

13.7 * 

0.024 

This year, 1 am able to cover the amount of instructional 
material my students need to learn 

74 

63 

10.9 * 

0.028 

This year, 1 am able cover the amount of instructional 
material my school expects 

82 

75 

6.5 

0.261 

Time Available for Instructional Activities 



Time available for enrichment activities 

68 

39 

29.3 ** 

0.002 

Time available for instruction in science 

81 

54 

26.0 ** 

0.001 

Time available for dedicated academic support 

64 

45 

19.5 * 

0.020 

Time available for students to pursue topics of interest 

44 

28 

16.3 * 

0.015 

Time available for instruction in ELA 

89 

74 

15.5 * 

0.010 

Time available for instruction in social studies 

71 

56 

15.2 

0.060 

Time available for instruction in math 

92 

78 

13.6 ** 

0.008 

Time available for physical activity for students 

41 

42 

-1.7 

0.808 

Time Available for Differentiated Instruction 



This year, 1 am able to differentiate instruction for students 
of multiple abilities 

89 

86 

2.9 

0.311 

This year, 1 am able to use a variety of instructional 
strategies (e.g., project-based learning, small-group 
learning) to teach my students 

92 

91 

1.0 

0.692 


EXHIBIT READS: Seventy-six percent of teachers in ELT schools indicated that students have sufficient time to learn about 
subjects, compared to an estimated 47 percent in the absence of ELT. The difference between the two groups was statistically 
significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Items 7w, 5f, 9c, 9b, lOe, 10c, lOf lOh, 10a, lOd, 
10b, lOg, 9d, 9a; Matched Comparison Teacher Surveys, Items 7p, 5f, 8c, 8b, 9e, 9c, 9f 9h, 9a, 9d, 9b, 9g, 8d, 8a. 

Sample: 1609 teachers. Nonresponse rates across ELT and matched comparison teacher survey items ranged from 6.7 to 24.6 
percent. 
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Exhibit E.4: Teacher Perception of Time Available for Planning and Professional Development 


Percent of 



Respondents 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


Time Available for Planning 


This year, this topic is a problem area: Amount of 
collaborative planning time 

46% 

66% 

-19.6 ** 

0.004 

Length of day allows coordination of instruction 

70 

53 

16.7 ** 

0.008 

Time available for collaborative planning 

57 

43 

14.1 * 

0.038 

This year, this topic is a problem area: Amount of 
individual planning time 

53 

60 

-6.6 

0.317 

Time available for individual planning time 

59 

55 

3.8 

0.532 

Time available for professional development 



Time available for professional development 

67 

72 

-4.8 

0.538 

This year, this topic is a problem area: Professional 
development opportunities 

41 

38 

3.4 

0.556 


EXHIBIT READS: Forty-six percent of teachers in ELT schools reported that the amount of time for collaborative planning 
was a problem area this year, compared to an estimated 66 percent in the absence of ELT. The difference between the two 
groups was statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Sun’eys of MA ELT School Teachers, Spring 2011, Items 6i, 5g, 32b, 6h, 32a, 32c, 6j; Matched 
Comparison Teacher Surveys, Items 6i, 5g, 31b, 6h, 31a, 31c, 6j. 

Sample: 1609 teachers. Nonresponse rates across ELT and matched comparison teacher survey items ranged from 7.0 to 13.4 
percent. 
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Exhibit E.5: Teacher Job Satisfaction 


Percent of 



Respondents 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


Satisfaction With Teaching 


1 plan to stay in the teaching profession until 1 retire 
If 1 could start over again, 1 would still become a teacher 
Overall, 1 am very satisfied with being a teacher 
Stress and challenges are not worth it 

89% 

88 

95 

15 

96% 

89 

96 

15 

-6.2 * 
-1.1 
-0.4 
0.2 

0.0325 

0.761 

0.764 

0.941 

Satisfaction With Teaching at This School 





Satisfied with salary 

61 

54 

7.1 

0.261 

Satisfied with teaching at this school 

87 

85 

1.5 

0.725 

Think about transferring 

34 

34 

-0.4 

0.935 


EXHIBIT READS: Eighty-nine percent of teachers in ELT schools reported that they planned to stay in the teaching 
profession until retirement, compared to an estimated 96 percent in the absence of ELT. The difference between the two groups 
was statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Items 5e, 5c, 5a, 5h, 5j, 5b, 5d; Matched 
Comparison Teacher Surveys, Items 5e, 5c, 5a, 5h, 5j, 5b, 5d. 

Sample: 1609 teachers. Nonresponse rates across ELT and matched comparison teacher survey items ranged from 5.5 to 13.8 
percent. 
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Exhibit E.6: Student and Teacher Relationships 



Percent of 
Respondents 




Actual 
ELT Mean 

Estimated 

Non-ELT 

Mean 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Student-Teacher Relationship 



1 get along well with my teachers 

59% 

65% 

-5.9 

0.283 

Students treat teachers with respect 1 

49 

44 

5.8 

0.472 

The teachers at my school care about the students 

84 

88 

-3.8 

0.412 

My teachers care about their students 

69 

73 

-3.7 

0.519 

None of my teachers know me 

12 

14 

-2.0 

0.478 

1 feel like 1 can talk to a teacher about my problems 

39 

41 

-1.7 

0.693 

1 know at least three teachers who 1 could ask for help 
with a problem 

58 

59 

-1.2 

0.798 

My teachers know a lot about me 

39 

39 

-0.6 

0.881 

This year, this topic is a problem area: Relationship with 
students 1 

9 

10 

-0.4 

0.862 

Teacher-Student Relationship 



Teachers and students spend sufficient instructional time 
together 1 

92 

68 

24.1 *** 

<.0001 

My teachers take time to explain what they are teaching 
so 1 understand it 

72 

76 

-4.2 

0.297 

1 get my questions answered in this class 

80 

83 

-2.5 

0.318 

1 have teachers who criticize me in class 

23 

22 

1.9 

0.461 

My teachers can tell if I’m not concentrating on school 
work 

62 

63 

-1.1 

0.754 

Adult-Student Relationship 



There are other people besides my regular teachers in the 
school who have time to give extra help 

48 

51 

-2.8 

0.390 

If 1 do not understand something in this class, there is 
someone in this school 1 can ask to explain it later 

75 

76 

-0.9 

0.718 

Peer Relationship 



Students at my school treat each other well 

43 

41 

2.4 

0.593 

Students treat each other with respect 1 

40 

38 

1.9 

0.793 


EXHIBIT READS: Fifty-nine percent of students in ELT schools indicated that they get along well with their teachers, 
compared to an estimated 65 percent in the absence of ELT. The difference between the two groups was not statistically 
significant. 

Notes: 1 Denotes an item from the teacher survey. All other items are from the student survey. 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Sun’eys of MA ELT School Teachers, Spring 2011, Items 7z, 6k, 7a, 7aa; Matched Comparison 
Teacher Surveys. Items 7s, 6k, 7a, 7t. Abt Associates’ Sun’eys ofMA ELT and Matched Comparison School Students, Spring 
2011, Items 11c, 81, lie, llj, lli, Ilk, llh, 11a, 9m, Ilf llg, 111, 9t, 8m. 

Sample: 1609 teachers, 4840 students. Nonresponse rates across ELT and matched comparison teacher survey items ranged 
from 4.6 to 6.0 percent. Nonresponse rates across ELT and matched comparison student survey items ranged from 1.7 to 4.1 
percent. 
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Exhibit E.7: Student Satisfaction and Engagement 


Percent of 



Respondents 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


General Satisfaction With School 


1 look forward to going to school 

54% 

66% 

-12.3 *** 

<.0001 

1 like being at my school 

56 

67 

-11.1 * 

0.037 

1 am bored in school 

55 

47 

8.4 

0.090 

1 get to do things at school that are fun and new 

67 

71 

-4.2 

0.347 

Kids at my school like being here 

33 

35 

-1.7 

0.755 

1 learn a lot in school 

91 

92 

-1.0 

0.599 

1 am getting a good education at my school 

88 

88 

0.3 

0.914 

Satisfaction With Teachers/Instruction 



My teachers help to make learning fun 

45 

53 

-7.6 

0.225 

1 think the teachers in my school would be better teachers 
if they had more time to teach their subjects 

36 

43 

-7.3 * 

0.048 

1 have trouble figuring out the answers in this class 

39 

33 

6.1 

0.072 

1 feel like my teachers go too slowly when they are 
teaching 

27 

31 

-4.9 

0.097 

1 would like school better if 1 had different teachers 

27 

24 

3.3 

0.292 

1 learn a lot in this class 

83 

86 

-2.4 

0.418 

The teacher of this class helps explain difficult concepts 
well 

82 

84 

-2.1 

0.545 

It matters to my teacher that all the students in this class 
work hard to learn 

89 

87 

1.9 

0.447 

My teacher knows a lot about the subject matter of this 
class 

93 

93 

0.2 

0.921 

Satisfaction With Length of School Day 



1 like the length of my school day this year 

22 

38 

-16.5 ** 

0.006 


EXHIBIT READS: Fifty-four percent of students in ELT schools indicated that they looked forward to going to school, 
compared to an estimated 66 percent in the absence of ELT. The difference between the two groups was statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Suiyeys of MA ELT and Matched Comparison School Students, Spring 2011, Items 8d, 8b, 8c, 8j, 8k, 

8a, 8e, lid, 8p, 9r, 8q, lib, 9q, 9n, 9u, 9p, 5. 

Sample: 4840 students. Nonresponse rates across ELT and matched comparison student survey items ranged from 1.1 to 3.2 
percent. 
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Exhibit E.8: Student Academic Engagement 



Percent of 
Respondents 




Actual 
ELT Mean 

Estimated 

Non-ELT 

Mean 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Academic Engagement 

This year, this topic is a problem area: Student academic 
performance 1 

68% 

83% 

-15.1 * 

0.011 

This year, this topic is a problem area: Homework 
completion rates 1 

70 

82 

-12.1 * 

0.028 

Students take their work seriously 1 

51 

41 

10.1 

0.144 

All of my classes are important to me 

65 

73 

-7.9 * 

0.032 

This year, this topic is a problem area: Student 
engagement 1 

67 

74 

-7.1 

0.210 

When 1 am in this class, 1 cannot wait for it to end 

55 

48 

6.4 

0.215 

If you had homework yesterday, did you do it? 

83 

89 

-5.8 

0.108 

1 get good grades in this class 

77 

83 

-5.8 

0.113 

1 check my work in this class for mistakes 

65 

71 

-5.6 

0.135 

1 pay attention in this class 

85 

89 

-4.1 

0.185 

1 am interested in the work 1 get to do in this class 

68 

71 

-3.0 

0.351 

1 try to get out of going to this class 

14 

16 

-2.3 

0.338 

Learning new things at school is important to me 

86 

88 

-1.3 

0.676 

When 1 am in this class, 1 only pretend to be working 

12 

13 

-0.9 

0.668 

My classes seem like a waste of time 

22 

22 

-0.7 

0.810 

1 try my best in this class 

86 

86 

0.3 

0.892 


EXHIBIT READS: Sixty-eight percent of teachers in ELT schools indicated that student academic performance was a 
problem area this year, compared to an estimated 83 percent in the absence of ELT. The difference between the two groups was 
statistically significant. 

Notes: 1 Denotes an item from the teacher survey. All other items are from the student survey. 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011, Items 6a, 6c, 7y, 6b; Matched Comparison Teacher- 
Surveys, Items 6a, 6c, 7r, 6b. Abt Associates’ Surveys ofMA ELT and Matched Comparison School Students, Spring 2011, 
Items 8i, 9s, 6b, 9i, 9h, 9c, 9a, 9j, 8o, 9e, 8f 9d. 

Sample: 1609 teachers, 4840 students. Nonresponse rates across ELT and matched comparison teacher survey items ranged 
from 7 to 19 percent. Nonresponse rates across ELT and matched comparison student survey items ranged from 1.6 to 3.1 
percent. 
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Exhibit E.9: Behavioral Engagement from Survey Items 



Percent of 
Respondents 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Actual 
ELT Mean 

Estimated 

Non-ELT 

Mean 

Behavioral Engagement 



This year, this topic is a problem area: Student behavior 1 

76% 

84% 

-7.4 

0.336 

1 follow the rules in this class 

87 

89 

-2.0 

0.452 

1 get in trouble in this class 

26 

25 

0.9 

0.706 

1 skip (cut) this class 

4 

5 

-0.8 

0.478 


EXHIBIT READS: Seventy-six percent of teachers in ELT schools indicated that student behavior was a problem area this 
year, compared to an estimated 84 percent in the absence of ELT. The difference between the two groups was not statistically 
significant. 

Notes: 1 Denotes an item from the teacher survey. All other items are from the student survey. 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Abt Associates’ Surveys of MA ELT School Teachers, Spring 2011. Item 6d; Matched Comparison Teacher Surveys, 
Item 6d. Abt Associates’ Sun’eys of MA ELT and Matched Comparison School Students, Spring 2011, Items 9k, 9b, 91. 

Sample: 1609 teachers, 4840 students. The nonresponse rate across ELT and matched comparison teachers for survey item 6d 
equaled 5.5 percent. Nonresponse rates across ELT and matched comparison student survey items ranged from 1.8 to 2.3 
percent. 
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Exhibit E.10: Behavioral Engagement from SIMS Data 


Percent of 



Respondents 




Estimated 


Statistical 

Actual 

Non-ELT 

Estimated 

Significance 

ELT Mean 

Mean 

Difference 

(p-value) 


Year 5 


Attendance rate 

93.88% 

94.37% 

-0.49 *** 

<0.001 

In-school suspension rate 

0.07 

-0.03 

0.10 *** 

<0.001 

Out-of-school suspension rate 

0.10 

0.06 

0.04 *** 

<0.001 

Truancy rate 

0.66 

0.53 

0.12 ** 

0.010 


EXHIBIT READS: At the end of the fifth year of ELT implementation, the average attendance rate at ELT schools was 93.88 
percent, compared to an estimated 94.37 percent in the absence of ELT. The difference between the two groups was statistically 
significant. 


Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE. 

Sample: Student records from ELT and matched comparison schools, cohorts 1-3 (n=49) from 2001-02 to 2010-11 for 
attendance and 2003-04 to 2010-11 for other measures: 257,348 records for attendance and 205,990 for other measures. Fewer 
than 1 percent of students were eliminated from the analytic sample due to missing covariates. 
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Appendix F. Exhibits for Chapter 6 


Exhibit F.1: Effect of ELT on Student and Teacher Characteristics, across Cohorts, after Five 


Years of Implementation 




Estimated 




Actual ELT 

Non-ELT 

Estimated 



Mean 

Mean 

Difference 

p-value 


Implementation Year 5 


Student Population 





Student enrollment 

574 

624 

-50.17 

0.355 

Percent low income 

77% 

84% 

-7.35 

0.054 

Percent minority 

77 

77 

-0.12 

0.971 

Percent male 

51 

52 

-1.14 

0.393 

Percent special education 

21 

21 

-0.38 

0.856 

Percent limited English proficient 

23 

18 

4.35 

0.283 

Percent first language not English 
Teacher Population 

42 

41 

1.4 

0.778 

Number of FTE Teachers 

45 

47 

-2.16 

0.555 

Percent of teachers licensed in their teaching 
assignment 

95 

99 

-3.93 

0.223 

Percent of core academic teachers highly 
qualified 

95 

98 

-3.3 

0.087 

Student-teacher ratio 

12 

12 

0.53 

0.459 


EXHIBIT READS: After the fifth year of ELT implementation, ELT schools had an average enrollment of 574 students, 
compared to an estimated 624 students in the absence of ELT. The difference was not statistically significant. 

Source: School-level data tables downloaded from the MA ESE website (http://profiles.doe.mass.edu). 

Sample: Full sample of 49 schools, including 24 ELT and 25 matched comparison schools. Student population outcomes are 
based on data from 2001-2002 to 2010-2011, including 496 records. Teacher population outcomes are based on data from 
2003-2004 to 2010-2011, including on 396 records. 


Exhibit F.2: Percent of Students who Attended the Same School in the Prior Year, after Five 


Years of Implementation 



Actual 
ELT Mean 

Actual 
MC Mean 

Statistical 

Significance 

(p-value) 

Implementation Year 5 

Percent of Students 

82% 

78% 

0.214 


EXHIBIT READS: After five years of ELT implementation, 82 percent of ELT students in non-entry grades attended the 
same school as in the previous year, compared to 78 percent of students in matched comparison schools. The difference 
between the two groups was not statistically significant 

Notes: Students whose grade level in the previous year was not available at their current school are excluded. P-values are 
from two-tailed independent samples t-tests that compare school-level mobility percentages by ELT status. 

* statistically significant at p < .05 level 

** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 
Source: Individual student records obtained from MA ESE. 

Sample: 6,433 students (3,448 ELT, 2,985 matched comparison). 
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Exhibit F.3: Effect of ELT on MCAS Subject/Grade Tests across Cohorts, by Implementation 
Year 



Actual 

Estimated 

Estimated 

Statistical 

Significance 


ELT Mean 

Non-ELT Mean 

Difference 

(p-value) 


Implementation Year 1 


Reading/ELA 





Grade 3 

-0.52 

-0.44 

-0.07 

0.542 

Grade 4 

-0.39 

-0.40 

0.01 

0.918 

Grade 7 

-0.43 

-0.47 

0.04 

0.559 

Math 





Grade 4 

-0.43 

-0.41 

-0.02 

0.808 

Grade 6 

-0.48 

-0.43 

-0.05 

0.396 

Grade 8 

-0.47 

-0.42 

-0.05 

0.350 

Science 





Grade 5 

-0.49 

-0.53 

0.04 

0.503 

Grade 8 

-0.55 

-0.55 

-0.00 

0.986 


Implementation Year 2 


Reading/ELA 





Grade 3 

-0.52 

-0.41 

-0.11 

0.321 

Grade 4 

-0.50 

-0.39 

-0.11 

0.211 

Grade 7 

-0.61 

-0.56 

-0.05 

0.565 

Math 





Grade 4 

-0.44 

-0.39 

-0.06 

0.405 

Grade 6 

-0.50 

-0.45 

-0.05 

0.355 

Grade 8 

-0.50 

-0.44 

-0.05 

0.548 

Science 





Grade 5 

-0.50 

-0.57 

0.08 

0.263 

Grade 8 

-0.74 

-0.69 

-0.05 

0.441 


Implementation Year 3 


Reading/ELA 





Grade 3 

-0.47 

-0.38 

-0.09 

0.317 

Grade 4 

-0.35 

-0.30 

-0.04 

0.703 

Grade 7 

-0.55 

-0.56 

0.01 

0.940 

Math 





Grade 4 

-0.38 

-0.33 

-0.05 

0.526 

Grade 6 

-0.48 

-0.51 

0.03 

0.741 

Grade 8 

-0.50 

-0.40 

-0.10 

0.499 

Science 





Grade 5 

-0.56 

-0.64 

0.09 

0.224 

Grade 8 

-0.66 

-0.64 

-0.03 

0.751 
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Exhibit F.3: Effect of ELT on MCAS Subject/Grade Tests across Cohorts, by Implementation 
Year 



Actual 

Estimated 

Estimated 

Statistical 

Significance 


ELT Mean 

Non-ELT Mean 

Difference 

(p-value) 


Implementation Year 4 


Reading/ELA 





Grade 3 

-0.48 

-0.51 

0.03 

0.811 

Grade 4 

-0.31 

-0.22 

-0.09 

0.614 

Grade 7 

-0.53 

-0.56 

0.03 

0.803 

Math 





Grade 4 

-0.26 

-0.30 

0.04 

0.708 

Grade 6 

-0.37 

-0.50 

0.13 

0.105 

Grade 8 

-0.36 

-0.33 

-0.03 

0.867 

Science 





Grade 5 

-0.30 

-0.58 

0.28*** 

<0.001 

Grade 8 

-0.60 

-0.70 

0.10 

0.324 


EXHIBIT READS: After the first year of ELT implementation students in ELT schools, on average, score 0.52 standard 
deviations below the statewide mean on the 3rd grade ELA portion of the MCAS test, as compared to 0.44 standard 
deviations below the mean in the absence of ELT. This difference was not statistically significant. 

Note: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE. 

Samples: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-2011. The sample 
sizes vary by cohort, implementation year, subject, and grade. The minimum number of schools is 29 and the maximum 
number of schools is 38. The minimum number of test records is 17,293 and the maximum number of test records is 39,242. 
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Exhibit F.4: Effect of ELT on MCAS Subject/Grade Tests after Five Years of Implementation 



Actual 

Estimated 

Estimated 

Statistical 

Significance 


ELT Mean 

Non-ELT Mean 

Difference 

(p-value) 


Implementation Year 5 


Reading/ELA 





Grade 3 

-0.57 

-0.49 

-0.09 

0.706 

Grade 4 

-0.50 

-0.29 

-0.21 

0.421 

Grade 7 

-0.64 

-0.73 

0.09 

0.648 

Math 





Grade 4 

-0.32 

-0.39 

0.07 

0.337 

Grade 6 

-0.38 

-0.57 

0.19 

0.171 

Grade 8 

-0.45 

-0.36 

-0.09 

0.630 

Science 





Grade 5 

-0.63 

-0.84 

0.21 

0.109 

Grade 8 

-0.75 

-0.69 

-0.06 

0.673 


EXHIBIT READS: After the five years of ELT implementation students in ELT schools, on average, score 0.57 standard 
deviations below the statewide mean on the 3rd grade ELA portion of the MCAS test, as compared to 0.49 standard 
deviations below the mean in the absence of ELT. This difference was not statistically significant. 

Note: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 
Source: Individual student records obtained from MA ESE. 

Samples: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-2011. The sample 
sizes vary by cohort, subject, and grade. The minimum number of schools is 29 and the maximum number of schools is 38. 
The minimum number of test records is 17,293 and the maximum number of test records is 39,242. 
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Exhibit F.5 presents findings from a sensitivity test conducted to check the robustness of the estimated 
effects of ELT on MCAS student achievement outcomes under a model specification that includes pre- 
ELT group achievement trends (ELT versus matched comparison) to account for possible group 
differences in any potential trends in the pre-ELT period, which would be captured by neither school 
fixed effects (since they are time-varying) nor year fixed effects (since they are not common to ELT and 
non-ELT schools). 

In the study’s Year Three report, tests for differences in pre-ELT group trends revealed that, across the 
majority of MCAS outcomes tested, the achievement of the ELT and matched comparison schools in the 
sample were on similar trajectories prior to the implementation of ELT. This suggested that the model 
specification used in analyses, which did not control for any potential differences in the pre-ELT group 
trends, were not likely to be affected by pre-ELT differences in achievement trends. This finding of no 
pre-ELT trend effects was confirmed in the year 4 outcomes report. Since the only change to the models 
in this report is the addition of one more post-ELT year of data, the estimates expected using the pre-ELT 
trend models will remain similar to the estimates produced in the main analytic models without pre-ELT 
trends. 

To be sure, the study directly tested whether there were differences in the pre-ELT group achievement 
trends, exploring whether and how the estimated effects of ELT differed under this alternative 
specification. This model specification differs from the primary model specification by the inclusion of a 
linear “time” variable that increases by 1 each year, an interaction between the time variable and an 
indicator for ELT status, and year fixed effects interacted with cohort status (as necessary) in the post- 
ELT years to constrain the fitted trends to pre-ELT years. 

Overall, the alternative specification does not change the conclusions drawn from the primary analytic 
model of no effect of ELT on the majority of the MCAS outcomes tested. There was not a statistically 
significant difference detected in the trends in performance of ELT and matched comparison schools 
among seven of the eight MCAS outcomes. While there does appear to be a statistically significant 
difference in the pre-ELT group trends of ELT and MC schools on the grade 4 ELA MCAS test, the result 
in terms of the effect of ELT on grade 4 ELA MCAS scores remains insignificant when the model 
controls for the pre-ELT group trend. The majority of estimated differences were similar in magnitude 
and direction across both approaches, with greater variation between the approaches evident within the 
math outcomes in general, and in the progression of implementation years. 
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Exhibit F.5: Model Specification Sensitivity Test-Results of ELT Effect on MCAS Subject/Grade 
Tests Estimated with Pre-ELT Group Trends, across Cohorts and by Implementation 
Year 


Model Features: 

Inclusion of Pre-ELT Group Trends 

Without Group Trends 


Actual 

ELT 

Mean 

Estimated 

Non-ELT 

Mean 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 

Pre-ELT 

Trend 

Difference 

Estimated 

Difference 

Statistical 

Significance 

(p-value) 


(D 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

Implementation Year 1 


Reading/ELA 
Grade 3 

-0.52 

-0.43 

-0.09 

0.565 

0 

-0.07 

0.542 

Grade 4 

-0.39 

-0.41 

0.02 

0.816 

-0.04* 

0.01 

0.918 

Grade 7 

-0.43 

-0.43 

-0.00 

0.987 

0 

0.04 

0.559 

Math 








Grade 4 

-0.43 

-0.39 

-0.04 

0.615 

-0.02 

-0.02 

0.808 

Grade 6 

-0.48 

-0.55 

0.07 

0.446 

-0.04 

-0.05 

0.396 

Grade 8 

-0.47 

-0.36 

-0.11 

0.098 

0 

-0.05 

0.350 

Science 








Grade 5 

-0.49 

-0.56 

0.07 

0.410 

-0.02 

0.04 

0.503 

Grade 8 

-0.55 

-0.55 

-0.00 

0.939 

-0.01 

-0.00 

0.986 

Implementation Year 2 


Reading/ELA 
Grade 3 

-0.52 

-0.37 

-0.15 

0.302 

0 

-0.11 

0.321 

Grade 4 

-0.50 

-0.50 

0.00 

0.987 

-0.04* 

-0.11 

0.211 

Grade 7 

-0.61 

-0.55 

-0.07 

0.567 

0 

-0.05 

0.565 

Math 








Grade 4 

-0.44 

-0.44 

-0.01 

0.957 

-0.02 

-0.06 

0.405 

Grade 6 

-0.50 

-0.58 

0.08 

0.439 

-0.04 

-0.05 

0.355 

Grade 8 

-0.50 

-0.39 

-0.11 

0.348 

0 

-0.05 

0.548 

Science 








Grade 5 

-0.50 

-0.61 

0.11 

0.196 

-0.02 

0.08 

0.263 

Grade 8 

-0.74 

-0.67 

-0.07 

0.336 

-0.01 

-0.05 

0.441 

Implementation Year 3 


Reading/ELA 
Grade 3 

-0.47 

-0.36 

-0.11 

0.418 

0 

-0.09 

0.317 

Grade 4 

-0.35 

-0.48 

0.14 

0.404 

-0.04* 

-0.04 

0.703 

Grade 7 

-0.55 

-0.57 

0.01 

0.901 

0 

0.01 

0.940 

Math 








Grade 4 

-0.38 

-0.43 

0.04 

0.722 

-0.02 

-0.05 

0.526 

Grade 6 

-0.48 

-0.70 

0.22 

0.101 

-0.04 

0.03 

0.741 

Grade 8 

-0.50 

-0.38 

-0.12 

0.424 

0 

-0.10 

0.499 

Science 








Grade 5 

-0.56 

-0.71 

0.15 

0.101 

-0.02 

0.09 

0.224 

Grade 8 

-0.66 

-0.68 

0.01 

0.894 

-0.01 

-0.03 

0.751 
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Exhibit F.5: Model Specification Sensitivity Test-Results of ELT Effect on MCAS Subject/Grade 
Tests Estimated with Pre-ELT Group Trends, across Cohorts and by Implementation 
Year 


Model Features: 

Inclusion of Pre-ELT Group Trends 

Without Group Trends 


Actual 

ELT 

Estimated 

Non-ELT 

Estimated 

Statistical 

Significance 

Pre-ELT 

Trend 

Estimated 

Statistical 

Significance 


Mean 

Mean 

Difference 

(p-value) 

Difference 

Difference 

(p-value) 


(D 

(2) 

(3) 

(4) 

(5) 

(6) 

(7) 

Implementation Year 4 


Reading/ELA 








Grade 3 

-0.48 

-0.49 

0.01 

0.972 

0 

0.03 

0.811 

Grade 4 

-0.31 

-0.45 

0.14 

0.568 

-0.04* 

-0.09 

0.614 

Grade 7 

-0.53 

-0.58 

0.05 

0.716 

0 

0.03 

0.803 

Math 








Grade 4 

-0.26 

-0.43 

0.17 

0.304 

-0.02 

0.04 

0.708 

Grade 6 

-0.37 

-0.73 

0.36* 

0.016* 

-0.04 

0.13 

0.105 

Grade 8 

-0.36 

-0.31 

-0.05 

0.781 

0 

-0.03 

0.867 

Science 








Grade 5 

-0.30 

-0.66 

0.36** 

0.001*** 

-0.02 

0.28*** 

<0.001 

Grade 8 

-0.60 

-0.76 

0.16 

0.140 

-0.01 

0.10 

0.324 

Implementation Year 5 


Reading/ELA 








Grade 3 

-0.57 

-0.47 

-0.10 

0.699 

0 

-0.09 

0.706 

Grade 4 

-0.50 

-0.57 

0.07 

0.828 

-0.04* 

-0.21 

0.421 

Grade 7 

-0.64 

-0.75 

0.11 

0.555 

0 

0.09 

0.648 

Math 








Grade 4 

-0.32 

-0.57 

0.24 

0.172 

-0.02 

0.07 

0.337 

Grade 6 

-0.38 

-0.84 

0.47* 

0.031 

-0.04 

0.19 

0.171 

Grade 8 

-0.45 

-0.34 

-0.10 

0.618 

0 

-0.09 

0.630 

Science 








Grade 5 

-0.63 

-0.94 

0.31 

0.060 

-0.02 

0.21 

0.109 

Grade 8 

-0.75 

-0.77 

0.02 

0.896 

-0.01 

-0.06 

0.673 


EXHIBIT READS: After the first year of ELT implementation students in ELT schools, on average, score 0.52 standard 
deviations below the statewide mean on the 3rd grade ELA portion of the MCAS test, as compared to 0.43 standard deviations 
below the mean in the absence of ELT, using a model specification that controls for trends in achievement before ELT in both 
ELT and matched comparison schools (columns 1 and 2). This difference was not statistically significant (column 3). The 
estimated difference in pre-ELT group trends between ELT and matched comparison schools was 0.00 standard deviations per 
year (column 5); not a statistically significant difference. In the primary analytic model, without the inclusion pre-ELT group 
trends, the estimated effect of ELT on the 3 rd grade ELA portion of the MCAS test was -0.07 standard deviations (column 6) and 
was not statistically significant (column 7). 

Note: 

* statistically significant at p< .05 level 
** statistically significant at p<.01 level 
*** statistically significant at p<.001 level. 

Source: Individual student records obtained from MA ESE. 

Samples: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-2011. The sample sizes 
vary by cohort, implementation year, subject, and grade. The minimum number of schools is 29 and the maximum number of 
schools is 38. The minimum number of test records is 17,293 and the maximum number of test records is 39,242. 
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Exhibit F.6 fits the model on a sample that includes all initial ELT schools in the model regardless of 
whether or not they continued to implement ELT in every year. This model relies on a sample that 
includes the three schools that discontinued ELT in 2009-10 as well as the three additional schools that 
discontinued ELT in 2010-1 1 and the matched comparison schools of these six schools. Including these 
schools in the sample made essentially no difference in terms of the estimated effects of ELT on MCAS 
results. 
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Exhibit F.6: Alternate Sample Sensitivity Test — Results of the Effect of ELT on MCAS 


Subject/Grade Tests Including Schools that Discontinued ELT, across Cohorts 
and by Implementation Year 



Actual 

Estimated 

Estimated 

Statistical 

Significance 


ELT Mean 

Non-ELT Mean 

Difference 

(p-value) 


Implementation Year 1 


Reading/ELA 





Grade 3 

-0.52 

-0.44 

-0.08 

0.524 

Grade 4 

-0.39 

-0.39 

-0.00 

0.988 

Grade 7 

-0.43 

-0.47 

0.04 

0.606 

Math 





Grade 4 

-0.43 

-0.42 

-0.01 

0.823 

Grade 6 

-0.48 

-0.43 

-0.05 

0.340 

Grade 8 

-0.47 

-0.42 

-0.06 

0.316 

Science 





Grade 5 

-0.49 

-0.54 

0.04 

0.468 

Grade 8 

-0.55 

-0.56 

0.00 

0.965 


Implementation Year 2 


Reading/ELA 





Grade 3 

-0.52 

-0.39 

-0.13 

0.279 

Grade 4 

-0.53 

-0.40 

-0.13 

0.087 

Grade 7 

-0.53 

-0.51 

-0.02 

0.829 

Math 





Grade 4 

-0.44 

-0.40 

-0.04 

0.517 

Grade 6 

-0.48 

-0.42 

-0.06 

0.249 

Grade 8 

-0.45 

-0.38 

-0.07 

0.398 

Science 





Grade 5 

-0.49 

-0.58 

0.09 

0.168 

Grade 8 

-0.63 

-0.60 

-0.03 

0.641 


Implementation Year 3 


Reading/ELA 





Grade 3 

-0.49 

-0.38 

-0.11 

0.194 

Grade 4 

-0.43 

-0.31 

-0.12 

0.220 

Grade 7 

-0.53 

-0.49 

-0.04 

0.641 

Math 





Grade 4 

-0.39 

-0.33 

-0.07 

0.379 

Grade 6 

-0.48 

-0.47 

-0.01 

0.853 

Grade 8 

-0.47 

-0.36 

-0.11 

0.391 

Science 





Grade 5 

-0.56 

-0.65 

0.10 

0.148 

Grade 8 

-0.59 

-0.58 

-0.01 

0.878 
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Exhibit F.6: Alternate Sample Sensitivity Test — Results of the Effect of ELT on MCAS 


Subject/Grade Tests Including Schools that Discontinued ELT, across Cohorts 
and by Implementation Year 



Actual 

Estimated 

Estimated 

Statistical 

Significance 


ELT Mean 

Non-ELT Mean 

Difference 

(p-value) 


Implementation Year 4 


Reading/ELA 





Grade 3 

-0.52 

-0.54 

0.01 

0.918 

Grade 4 

-0.36 

-0.29 

-0.08 

0.633 

Grade 7 

-0.53 

-0.57 

0.04 

0.752 

Math 





Grade 4 

-0.34 

-0.34 

0.00 

0.991 

Grade 6 

-0.38 

-0.51 

0.13 

0.083 

Grade 8 

-0.36 

-0.34 

-0.02 

0.889 

Science 





Grade 5 

-0.43 

-0.71 

0.28** 

0.001 

Grade 8 

-0.60 

-0.72 

0.12 

0.249 


Implementation Year 5 


Reading/ELA 





Grade 3 

-0.57 

-0.48 

-0.09 

0.681 

Grade 4 

-0.50 

-0.30 

-0.20 

0.432 

Grade 7 

-0.64 

-0.71 

0.07 

0.681 

Math 





Grade 4 

-0.32 

-0.36 

0.03 

0.660 

Grade 6 

-0.38 

-0.56 

0.18 

0.193 

Grade 8 

-0.45 

-0.37 

-0.08 

0.683 

Science 





Grade 5 

-0.63 

-0.84 

0.21 

0.111 

Grade 8 

-0.75 

-0.73 

-0.02 

0.860 


EXHIBIT READS: After the first year of ELT implementation students in ELT schools, on average, score .52 standard 
deviations below the statewide mean on the 3rd grade ELA portion of the MCAS test, as compared to .44 standard deviations 
below the mean in the absence of ELT. This difference was not statistically significant. 

Note: 

* statistically significant at p < .05 level 

** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE. 

Samples: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-2011. The sample 
sizes vary by cohort, implementation year, subject, and grade. The minimum number of schools is 29 and the maximum 
number of schools is 38. The minimum number of test records is 17,804 and the maximum number of test records is 41,027. 
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Exhibit F.7 fits the models with only five and only three years of pre-ELT implementation data for each 
cohort, using both the primary model specification and the alternative specification including pre-ELT 
group trends in achievement. 

The primary analytic sample was constructed using all available data, which included MCAS test score 
data beginning in 2002 (or 2003 for science outcomes), regardless of cohort. Because the ELT program 
had a staggered implementation, with new schools added in subsequent years to a second and then third 
cohort, this meant that there were different numbers of years of pre-ELT data available for different 
cohorts: up to five years for Cohort 1, six years for Cohort 2, and seven years for Cohort 3. With no 
consistent recommendation in the literature about the number of pre-program years of data necessary or 
reasonable to use for fitting the interrupted time-series model used for analysis, the study tested the effect 
of limiting the sample to five and three years of pre -program data for all cohorts. This sample was tested 
using both the primary analytic model, and the additional model (discussed above) that included terms to 
fit differential pre-ELT group trends for the ELT and matched comparison schools. 

Consistent with the other specification and sensitivity tests, the overall findings from these tests do not 
call into question the findings of no estimated effects of ELT on the MCAS outcomes tested, except for 
grade 5 science. Using the primary model without fitted pre-ELT group trends, the results from each 
alternate sample are extremely consistent with the original results (column 1). The only difference is that 
in the model where the sample is confined to three pre-ELT years of data (column 3), the effect of ELT 
on grade 6 math MCAS scores reaches statistical significance after four years of ELT implementation. 

There is generally greater variation from these samples when tested using the model including pre-ELT 
group- specific trends, and the statistically significant result found in the model without trends using only 
three years of pre-ELT data (column 3) is mirrored in the results for the model with pre-ELT group trends 
that uses the analytic sample (column 4). In all model specifications and using all sample configurations 
except for the pre-ELT group trend model that confines the sample to three years prior to ELT (column 6) 
there is a statistically significant effect of ELT on grade 5 science MCAS scores after four years of ELT 
implementation. 
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Exhibit F.7: Alternate Samples Sensitivity Test — Results of the Effect of ELT on MCAS Subject/ 


Grade Tests when Sample is Reduced to Only Five or Three Years of Pre-ELT 
Implementation Data for each Cohort, across Cohorts and by Implementation Year 


Model Features: 

No Pre-ELT Trends 

Including Pre-ELT Trends 


Analytic 

Sample 

Estimate 

Using 5 
Years Pre- 
ELT Data 

Using 3 
Years Pre- 
ELT Data 

Analytic 

Sample 

Estimate 

Using 5 
Years Pre- 
ELT Data 

Using 3 
Years Pre- 
ELT Data 


(D 

(2) 

(3) 

(5) 

(6) 

(7) 

Implementation Year 1 




Reading/ELA 
Grade 3 

-0.07 

-0.07 

-0.07 

-0.09 

-0.09 

-0.13 

Grade 4 

0.01 

0.02 

0.03 

0.02 

0.02 

0.08 

Grade 7 

0.04 

0.05 

0.05 

-0.00 

-0.01 

-0.04 

Math 







Grade 4 

-0.02 

-0.00 

-0.02 

-0.04 

-0.06 

0.04 

Grade 6 

-0.05 

-0.03 

-0.02 

0.07 

0.06 

0.08 

Grade 8 

-0.05 

-0.05 

-0.03 

-0.11 

-0.09 

-0.06 

Science 







Grade 5 

0.04 

0.04 

0.04 

0.07 

0.10 

0.11 

Grade 8 

-0.00 

0.00 

0.02 

-0.00 

0.02 

-0.03 

Implementation Year 2 


Reading/ELA 
Grade 3 

-0.11 

-0.11 

-0.12 

-0.15 

-0.15 

-0.23 

Grade 4 

-0.11 

-0.10 

-0.10 

0.00 

-0.00 

0.08 

Grade 7 

-0.05 

-0.04 

-0.05 

-0.07 

-0.07 

-0.12 

Math 







Grade 4 

-0.06 

-0.04 

-0.07 

-0.01 

-0.03 

0.10 

Grade 6 

-0.05 

-0.04 

-0.02 

0.08 

0.08 

0.11 

Grade 8 

-0.05 

-0.04 

-0.02 

-0.11 

-0.08 

-0.01 

Science 







Grade 5 

0.08 

0.07 

0.07 

0.11 

0.16 

0.16 

Grade 8 

-0.05 

-0.05 

-0.03 

-0.07 

-0.03 

-0.11 

Implementation Year 3 


Reading/ELA 
Grade 3 

-0.09 

-0.09 

-0.10 

-0.11 

-0.10 

-0.23 

Grade 4 

-0.04 

-0.04 

-0.04 

0.14 

0.12 

0.22 

Grade 7 

0.01 

0.02 

0.02 

0.01 

0.02 

-0.04 

Math 







Grade 4 

-0.05 

-0.05 

-0.08 

0.04 

0.01 

0.16 

Grade 6 

0.03 

0.04 

0.06 

0.22 

0.21 

0.25 

Grade 8 

-0.10 

-0.08 

-0.05 

-0.12 

-0.07 

0.05 

Science 







Grade 5 

0.09 

0.07 

0.07 

0.15 

0.21 

0.19 

Grade 8 

-0.03 

-0.02 

-0.00 

0.01 

0.06 

-0.02 
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Exhibit F.7: Alternate Samples Sensitivity Test — Results of the Effect of ELT on MCAS Subject/ 


Grade Tests when Sample is Reduced to Only Five or Three Years of Pre-ELT 
Implementation Data for each Cohort, across Cohorts and by Implementation Year 


Model Features: 

No Pre-ELT Trends 

Including Pre-ELT Trends 


Analytic 

Sample 

Estimate 

Using 5 
Years Pre- 
ELT Data 

Using 3 
Years Pre- 
ELT Data 

Analytic 

Sample 

Estimate 

Using 5 
Years Pre- 
ELT Data 

Using 3 
Years Pre- 
ELT Data 


(D 

(2) 

(3) 

(5) 

(6) 

(7) 

Implementation Year 4 


Reading/ELA 
Grade 3 

0.03 

0.03 

0.02 

0.01 

0.02 

-0.14 

Grade 4 

-0.09 

-0.09 

-0.11 

0.14 

0.13 

0.23 

Grade 7 

0.03 

0.05 

0.05 

0.05 

0.07 

-0.01 

Math 







Grade 4 

0.04 

0.03 

-0.03 

0.17 

0.12 

0.30 

Grade 6 

0.13 

0.14 

0.16* 

0.36* 

0.35 

0.40 

Grade 8 

-0.03 

0.00 

0.04 

-0.05 

0.04 

0.18 

Science 







Grade 5 

0.28*** 

0.27*** 

0.23** 

0.36** 

0.43** 

0.39 

Grade 8 

0.10 

0.11 

0.13 

0.16 

0.22 

0.11 

Implementation Year 5 


Reading/ELA 
Grade 3 

-0.09 

-0.08 

-0.08 

-0.10 

-0.07 

-0.25 

Grade 4 

-0.21 

-0.23 

-0.26 

0.07 

0.07 

0.18 

Grade 7 

0.09 

0.11 

0.12 

0.11 

0.14 

0.06 

Math 







Grade 4 

0.07 

0.06 

-0.00 

0.24 

0.18 

0.41 

Grade 6 

0.19 

0.21 

0.23 

0.47* 

0.45 

0.52 

Grade 8 

-0.09 

-0.05 

-0.01 

-0.10 

-0.00 

0.17 

Science 







Grade 5 

0.21 

0.19 

0.12 

0.31 

0.39* 

0.32 

Grade 8 

-0.06 

-0.05 

-0.02 

0.02 

0.10 

-0.03 


EXHIBIT READS: At the end of the first year of ELT implementation, the estimated effect of ELT on 3rd grade ELA MCAS 
scores was -0.07 standard deviations, which was not statistically significant, using a model without pre-ELT trends. Confining each 
cohort to five and three years of pre-ELT data, using the same model, produced the same estimated effect of -0.07 standard 
deviations, which again was not statistically significant. Using a model specification including pre-ELT trends, at the end of the 
first year of ELT implementation, the estimated effect of ELT on grade 3 ELA scores was -0.09 standard deviations; the effect was 
not statistically significant. Using the same model and confining the sample to five and three years of pre-ELT trend data for each 
cohort produced estimated effects of -0.09 and -0.13 standard deviations, respectively, neither of which was statistically significant. 
Notes: Columns 2 and 5, labeled “Using 5 Years Pre-ELT Data,” indicate that the sample was limited to only five years of pre-ELT 
implementation data for each cohort; columns 3 and 6, labeled “Using 3 Years of Pre-ELT Data,” indicate that the sample was 
limited to only three years of pre-ELT implementation data for each cohort. Models in columns 1-3, labeled "No Pre-ELT Trends,” 
fit a specification identical to the model used to generate primary estimates of the effect of ELT on MCAS scores (see exhibit E.3); 
models in columns 4 to 6, labeled “Including Pre-ELT Trends” fit a specification that included additional terms to allow for the pre- 
ELT implementation MCAS trends to differ between ELT and matched comparison schools (see exhibit F.5). 

* statistically significant at p< .05 level 
** statistically significant at pc.Ol level 
*** statistically significant at pc.OOl level. 

Source: Individual student records obtained from MA ESE. 

Samples: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-2011. The sample sizes 
vary by model specification, cohort, implementation year, subject, and grade. The minimum number of schools is 29 and the 
maximum number of schools is 38. The minimum number of test records is 12,1 1 1 and the maximum number of test records is 
39,242. 
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Exhibit F.8: Difference in the Effect of ELT on MCAS Subject/Grade Tests between Higher and 
Lower Implementing Schools after Five Years of Implementation 
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EXHIBIT READS: After five years of ELT implementation, the effect of ELT in higher implementing schools on grade 3 
ELA MCAS scores was, on average, 0.30 standard deviations higher than the effect of ELT on grade 3 ELA MCAS scores in 
lower implementing schools. This difference was not statistically significant. 

Notes: 

* statistically significant at p < .05 level 
** statistically significant at p < .01 level 
*** statistically significant at p < .001 level 

Source: Individual student records obtained from MA ESE; Abt Associates' Interviews ofMA ELT and Matched Comparison 
School Principals, Spring 2011; Abt Associates' Surveys ofMA ELT and Matched Comparison School Teachers, Spring 
2011 . 

Sample: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-2011. Only ELT 
schools and their matched comparison schools with full data on the implementation index are included in these analyses. The 
sample sizes vary by cohort, subject, and grade. The minimum number of schools is 7 and the maximum number of schools is 
14. The minimum number of test records is 4,821 and the maximum number of test records is 13,995. 
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Exhibit F.9: Effect of ELT on MCAS Subject/Grade Tests between Higher and Lower 
Implementing Schools across Cohorts, by Implementation Year 



Higher Implementing 
Schools 

Lower Implementing 
Schools 

Difference between 
Higher and Lower 
Implementing Schools 


Estimated 

ELT 

Effect 

Statistical 

Significance 

(p-value) 

Estimated 

ELT 

Effect 

Statistical 

Significance 

(p-value) 

Estimated 

ELT 

Effect 

Statistical 

Significance 

(p-value) 

Implementation Year 1 






Reading/ELA 
Grade 3 

-0.038 

0.61 

0.12 

0.234 

-0.16 

0.229 

Grade 4 

0.12 

0.345 

0.04 

0.596 

0.08 

0.599 

Grade 7 

0.069 

0.123 

0.17 

0.070 

-0.10 

0.362 

Math 







Grade 4 

0.083 

0.466 

-0.02 

0.850 

0.10 

0.501 

Grade 6 

-0.019 

0.797 

-0.01 

0.883 

-0.01 

0.955 

Grade 8 

-0.074 

0.128 

-0.09 

0.436 

0.01 

0.920 

Science 







Grade 5 

0.142 

0.234 

0.01 

0.799 

0.13 

0.316 

Grade 8 

-0.111 

0.119 

0.03 

0.687 

-0.14 

0.187 

Implementation Year 2 






Reading/ELA 
Grade 3 

-0.06 

0.481 

0.19 

0.218 

-0.25 

0.179 

Grade 4 

-0.182 

0.277 

-0.08 

0.666 

-0.11 

0.673 

Grade 7 

-0.046 

0.589 

0.07 

0.239 

-0.12 

0.278 

Math 







Grade 4 

-0.059 

0.503 

0.00 

0.988 

-0.06 

0.664 

Grade 6 

-0.165 

0.115 

0.02 

0.727 

-0.18 

0.134 

Grade 8 

0.118 

0.173 

-0.12 

0.201 

0.24 

0.078 

Science 







Grade 5 

0.258 

0.001*** 

0.07 

0.340 

0.19 

0.099 

Grade 8 

0.05 

0.657 

-0.07 

0.446 

0.12 

0.419 

Implementation Year 3 






Reading/ELA 
Grade 3 

-0.173 

0.037* 

-0.01 

0.942 

-0.16 

0.383 

Grade 4 

0.041 

0.786 

-0.01 

0.964 

0.05 

0.842 

Grade 7 

0.035 

0.744 

0.09 

0.245 

-0.05 

0.705 

Math 







Grade 4 

0.052 

0.561 

0.10 

0.415 

-0.05 

0.761 

Grade 6 

0.025 

0.835 

-0.01 

0.885 

0.04 

0.803 

Grade 8 

0.168 

0.24 

-0.24 

0.011* 

0.41 

0.028* 

Science 







Grade 5 

0.176 

0.121 

0.10 

0.062 

0.08 

0.521 

Grade 8 

0.078 

0.362 

-0.07 

0.471 

0.15 

0.265 
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Exhibit F.9: Effect of ELT on MCAS Subject/Grade Tests between Higher and Lower 
Implementing Schools across Cohorts, by Implementation Year 



Higher Implementing 
Schools 

Lower Implementing 
Schools 

Difference between 
Higher and Lower 
Implementing Schools 


Estimated 

Statistical 

Estimated 

Statistical 

Estimated 

Statistical 


ELT 

Significance 

ELT 

Significance 

ELT 

Significance 


Effect 

(p-value) 

Effect 

(p-value) 

Effect 

(p-value) 

Implementation Year 4 






Reading/ELA 







Grade 3 

-0.133 

0.194 

0.06 

0.774 

-0.20 

0.428 

Grade 4 

-0.077 

0.751 

-0.21 

0.374 

0.13 

0.703 

Grade 7 

0.069 

0.645 

0.05 

0.573 

0.02 

0.895 

Math 







Grade 4 

0.162 

0.161 

-0.06 

0.588 

0.23 

0.190 

Grade 6 

0.099 

0.323 

0.16 

0.069 

-0.06 

0.644 

Grade 8 

0.302 

0.001*** 

-0.21 

0.048* 

0.51 

0.002** 

Science 







Grade 5 

0.398 

0.005** 

0.18 

0.014* 

0.22 

0.185 

Grade 8 

0.141 

0.002** 

0.08 

0.461 

0.07 

0.556 ( 

Implementation Year 5 






Reading/ELA 







Grade 3 

0.007 

0.937 

-0.29 

0.122 

0.30 

0.173 

Grade 4 

0.133 

0.384 

-0.54 

<0.001*** 

0.67 

0.008** 

Grade 7 

0.266 

0.145 

0.02 

0.804 

0.25 

0.219 

Math 







Grade 4 

0.186 

0.012* 

-0.14 

0.176 

0.33 

0.02* 

Grade 6 

0.297 

0.165 

0.16 

0.069 

0.13 

0.573 

Grade 8 

0.266 

0.002** 

-0.39 

<0.001*** 

0.66 

<0.001*** 

Science 







Grade 5 

0.611 

<0.001*** 

-0.03 

0.495 

0.64 

<0.001*** 

Grade 8 

0.204 

0.01** 

-0.30 

<0.001*** 

0.50 

<0.001*** 


EXHIBIT READS: After the first year of ELT implementation, students in higher implementing schools scored, on average, 
0.38 standard deviations below what they were predicted to score in the absence of ELT; students in lower implementing 
schools scored, on average, 0.12 standard deviations above what they were predicted to score in the absence of ELT; the effect 
of ELT in high implementing schools on grade 3 ELA MCAS scores was, on average, 0.16 standard deviations lower than the 
effect of ELT on grade 3 ELA MCAS scores in lower implementing schools. These differences were not statistically significant. 

Notes: 

* statistically significant at p< .05 level 
** statistically significant at p<.01 level 
*** statistically significant atp<.001 level. 

Source: Individual student records obtained from MA ESE ; Abt Associates’ Interviews ofMA ELT and Matched Comparison 
School Principals, Spring 2011; Abt Associates’ Surveys ofMA ELT and Matched Comparison School Teachers, Spring 2011. 

Sample: Student MCAS records from ELT and matched comparison schools from 2001-2002 to 2010-201 1 . Only ELT schools 
and their matched comparison schools with full data on the implementation index are included in these analyses. The sample 
sizes vary by cohort, subject, and grade. The minimum number of schools is 7 and the maximum number of schools is 14. The 
minimum number of test records is 4,821 and the maximum number of test records is 13,995. 
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Exhibit F.10: Median Student Growth Percentile (SGP) 2011 MCAS Scores and ELT 
Implementation Index Scores, ELT Schools 


Panel 1: ELA 



EXHIBIT READS: The grade 4 ELA graph indicates, for each ELT school, the median SGP on the 4th grade ELA portion of the MCAS test 
in 2010-1 1 (y-axis) and the school's total score on the implementation index developed for the study (x-axis). 

Note: Each point on each graph represents a single ELT school. MA ESE does not calculate SGPs for science MCAS tests or for 3rd grade 
tests. 

Source: Individual student records obtained from MA ESE; Abt Associates’ Interviews ofMA ELT School Principcds, Spring 2011; Abt 
Associates’ Surveys ofMA ELT School Teachers, Spring 2011. 

Sample: Current ELT schools as of the 2010-11 school year. 
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Exhibit F.10: Median Student Growth Percentile (SGP) 2011 MCAS Scores and ELT 
Implementation Index Scores, ELT Schools 

Panel 2: Math 
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EXHIBIT READS: The grade 4 Math graph indicates, for each ELT school, the median SGP on the 4th grade Math portion of the MCAS test 
in 2010-1 1 (y-axis) and the school's total score on the implementation index developed for the study (x-axis). 

Note: Each point on each graph represents a single ELT school. MA ESE does not calculate SGPs for science MCAS tests or for 3rd grade 
tests. 

Source: Individual student records obtained from MA ESE; Abt Associates’ Interviews ofMA ELT School Principals, Spring 2011; Abt 
Associates’ Surveys ofMA ELT School Teachers, Spring 2011. 

Sample: Current ELT schools as of the 2010-1 1 school year. 
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